From 589b4d04080fd282e2119cf0d448367ad126cd9e Mon Sep 17 00:00:00 2001 From: Denys Melnyk Date: Wed, 10 Jun 2026 16:58:24 +0300 Subject: [PATCH 1/3] Consolidate Argo CD gateway troubleshooting and add install prerequisites - Move the TLS errors content from the instances page into the central troubleshooting page, leaving a pointer behind so the existing oc.to/argo-cd-gateway-troubleshooting anchor still lands somewhere useful - Document the "registered but failed to connect to Octopus" install failure (gRPC port 8443 unreachable), including the orphaned gateway left behind by an atomic helm install rollback - Add a Prerequisites section with the gateway's network requirements --- src/pages/docs/argo-cd/instances/index.md | 76 ++++---------- src/pages/docs/argo-cd/troubleshooting.md | 118 ++++++++++++++++++++-- 2 files changed, 132 insertions(+), 62 deletions(-) diff --git a/src/pages/docs/argo-cd/instances/index.md b/src/pages/docs/argo-cd/instances/index.md index a0b6d03531..876b605091 100644 --- a/src/pages/docs/argo-cd/instances/index.md +++ b/src/pages/docs/argo-cd/instances/index.md @@ -1,7 +1,7 @@ --- layout: src/layouts/Default.astro pubDate: 2025-09-15 -modDate: 2025-12-08 +modDate: 2026-06-10 navSection: Argo CD Instances navTitle: Overview title: Overview @@ -29,6 +29,24 @@ The [Kubernetes agent](/docs/kubernetes/targets/kubernetes-agent) and the [Kuber ::: +## Prerequisites + +The gateway makes three outgoing connections. Before installing, make sure all of them are reachable from inside your cluster: + +| The gateway must reach | How | +| --- | --- | +| Octopus Server REST API | HTTPS, port `443` | +| Octopus Server gRPC endpoint | gRPC (HTTP/2), port `8443` by default | +| Argo CD API server | In-cluster gRPC, e.g. `..svc.cluster.local` | + +:::div{.warning} +If your Octopus Server sits behind a load balancer, proxy, or firewall, make sure the gRPC port (`8443` by default) is forwarded to Octopus Server and the proxy supports HTTP/2. Forwarding only HTTPS (`443`) is a common cause of installation failure, where the gateway registers successfully but never connects. See [Troubleshooting](/docs/argo-cd/troubleshooting#argo-cd-gateway-registers-but-fails-to-connect-to-octopus-server) for details. +::: + +:::div{.hint} +The gateway holds long-lived gRPC streams and sends a keep-alive every 30 seconds by default. If a load balancer between the cluster and Octopus Server closes idle connections, set its idle timeout to comfortably exceed the keep-alive interval (`gateway.octopus.keepAlive.intervalSeconds`). +::: + ## Installing the Octopus Argo CD Gateway The gateway is installed using [Helm](https://helm.sh) via the [octopusdeploy/octopus-argocd-gateway-chart](https://hub.docker.com/r/octopusdeploy/octopus-argocd-gateway-chart) chart. @@ -206,61 +224,7 @@ The Octopus Argo CD gateway Helm chart follows [Semantic Versioning](https://sem ## Troubleshooting -### Argo CD TLS Errors - -If your gateway is unable to connect to your Argo CD instance due to TLS errors it is likely due to the certificate that Argo CD is serving traffic with. - -#### Self Signed Certificate - -If you are getting an error that looks like this: - -```text -tls: failed to verify certificate: x509: certificate signed by unknown authority -``` - -It is most likely due to Argo CD using a self-signed certificate, if it is intended that your certificate is self-signed you can disable certificate verification by doing the following: - -Using Helm for existing installation: - -```bash -helm upgrade --atomic \ ---version "1.0.0" \ ---namespace "{{GATEWAY_NAMESPACE}}" \ ---reset-then-reuse-values \ ---set gateway.argocd.insecure="true" \ ---set gateway.argocd.plaintext="false" \ -{{EXISTING_HELM_RELEASE_NAME}} \ -oci://registry-1.docker.io/octopusdeploy/octopus-argocd-gateway-chart -``` - -:::div{.warning} -By setting `gateway.argocd.insecure="true"`, TLS certificate verification will no longer be performed between the gateway and the Argo CD instance. Make sure this configuration is necessary to avoid potential security issues. -::: - -#### No Certificate - -If you are running your Argo CD instance without a certificate due to terminating SSL at a load balancer level the gateway will likely fail to connect with the following error: - -```text -transport: authentication handshake failed: EOF -``` - -This is because the gateway is configured by default to require encrypted traffic, if it is intended that you don't have a certificate you can disable encryption between the gateway and Argo CD by doing the following: - -```bash -helm upgrade --atomic \ ---version "1.0.0" \ ---namespace "{{GATEWAY_NAMESPACE}}" \ ---reset-then-reuse-values \ ---set gateway.argocd.insecure="false" \ ---set gateway.argocd.plaintext="true" \ -{{EXISTING_HELM_RELEASE_NAME}} \ -oci://registry-1.docker.io/octopusdeploy/octopus-argocd-gateway-chart -``` - -:::div{.warning} -By setting `gateway.argocd.plaintext="true"`, all traffic between the gateway and Argo CD will be unencrypted. Make sure this configuration is necessary to avoid potential security issues. -::: +If your gateway is unable to connect to your Argo CD instance or Octopus Server (e.g. due to TLS errors), see [Troubleshooting Argo CD in Octopus](/docs/argo-cd/troubleshooting) for common issues and resolutions. ## Deleting an Octopus Argo CD Gateway diff --git a/src/pages/docs/argo-cd/troubleshooting.md b/src/pages/docs/argo-cd/troubleshooting.md index 6b5f3204e3..8f3fffec1a 100644 --- a/src/pages/docs/argo-cd/troubleshooting.md +++ b/src/pages/docs/argo-cd/troubleshooting.md @@ -1,7 +1,7 @@ --- layout: src/layouts/Default.astro pubDate: 2025-09-15 -modDate: 2025-09-15 +modDate: 2026-06-10 title: Troubleshooting Argo CD in Octopus navTitle: Troubleshooting description: How to resolve configuration issues @@ -38,14 +38,45 @@ Resolution: ### Argo CD Gateway install fails initial health check +#### Failed to connect to Octopus + Behavior: - Install Argo CD Gateway dialog states: - - "established a connection" was successful - - Health check failed -- The Gateway pod is in a CrashLoopBackoff + - "Gateway registered with Octopus" was successful + - "Failed to connect to Octopus" and "Failed to connect to Argo CD" both show as failed +- The gateway pod is in a CrashLoopBackoff +- In a Kubernetes viewer (e.g. K9s), the gateway pod logs state "*Gateway failed to connect to Octopus*" +- If installed with `helm install --atomic`, the install fails and rolls back, removing the gateway from the cluster. The registered gateway still appears under **Infrastructure ➜ Argo CD Instances** but will never become healthy + +Cause: + +- The gateway cannot establish a gRPC connection to Octopus Server. Both "Failed to connect" rows in the dialog are caused by this single problem, not two separate ones +- Registration uses the REST API url (`registration.octopus.serverApiUrl`), while the running gateway connects to a separate gRPC endpoint (`gateway.octopus.serverGrpcUrl`, port `8443` by default), so a successful registration does not mean the gRPC endpoint is reachable + +Resolution: + +- Confirm port `8443` is open and routed through to Octopus Server. A load balancer, proxy, or firewall that only forwards HTTPS (`443`) is a common cause. Probe it from inside the cluster: + +```bash +kubectl run port-check --image=busybox --restart=Never --rm -it -- \ + sh -c 'nc -z -w 5 your-octopus-url 8443 && echo REACHABLE || echo UNREACHABLE' +``` + +- Confirm `gateway.octopus.serverGrpcUrl` points at your Octopus Server's gRPC endpoint, including the port (not the web url) +- If the gateway logs a certificate thumbprint mismatch, confirm `gateway.octopus.serverThumbprint` matches your Octopus Server's certificate thumbprint +- Inspect the gateway pod logs for connection details: `kubectl logs deploy/octopus-argocd-gateway -n ` +- If the install was rolled back (e.g. `helm install --atomic` failed and cleaned up the cluster), delete the orphaned Argo CD Gateway in Octopus, resolve the connection issue, and re-run the installation + +#### Failed to connect to ArgoCD + +Behavior: + +- Install Argo CD Gateway dialog states: + - "Gateway registered with Octopus" was successful + - "Failed to connect to Argo CD" show as failed - In a Kubernetes viewer (e.g. K9s), the gateway pod logs state "*error validating connection to Argo CD*" -- In Octopus, the healthcheck task log contains: "The Argo CD Gateway has not established a gRPC connection to Octopus Server" +- In Octopus when navigating to newly added ArgoCD instance "Gateway connectivity" tab show "Argo CD Connectivity Issues" warning Cause: @@ -54,10 +85,85 @@ Cause: Resolution: - Confirm the URL specified for the `gateway.argocd.serverGrpcUrl` matches the expected grpc endpoint of your argo instance (`..svc.cluster.local`) -- If your Argo CD instance is using a self-signed certificate ensure `gateway.argocd.insecure` is set to `true` +- If your Argo CD instance is using a self-signed certificate ensure `gateway.argocd.insecure` is set to `true` (see [TLS errors](#argo-cd-gateway-cannot-connect-to-argo-cd-due-to-tls-errors) below) - If your Argo CD instance is running in "insecure" mode, ensure `gateway.argocd.plaintext` is set to `true` (false otherwise) - In Octopus, delete the registered Argo CD Gateway, follow all required helm deletion commands, and reinstall +### Argo CD Gateway cannot connect to Argo CD due to TLS errors + +If your gateway is unable to connect to your Argo CD instance due to TLS errors it is likely due to the certificate that Argo CD is serving traffic with. + +#### Self Signed Certificate + +Behavior: + +- The gateway is unable to connect to your Argo CD instance +- The gateway pod logs contain: + +```text +tls: failed to verify certificate: x509: certificate signed by unknown authority +``` + +Cause: + +- Argo CD is using a self-signed certificate + +Resolution: + +- Configure the gateway to trust your certificate, as described in [Trusting Certificates](/docs/argo-cd/instances#trusting-certificates) +- Alternatively, if it is intended that your certificate is self-signed, you can disable certificate verification by doing the following: + +Using Helm for existing installation: + +```bash +helm upgrade --atomic \ +--version "1.0.0" \ +--namespace "{{GATEWAY_NAMESPACE}}" \ +--reset-then-reuse-values \ +--set gateway.argocd.insecure="true" \ +--set gateway.argocd.plaintext="false" \ +{{EXISTING_HELM_RELEASE_NAME}} \ +oci://registry-1.docker.io/octopusdeploy/octopus-argocd-gateway-chart +``` + +:::div{.warning} +By setting `gateway.argocd.insecure="true"`, TLS certificate verification will no longer be performed between the gateway and the Argo CD instance. Make sure this configuration is necessary to avoid potential security issues. +::: + +#### No Certificate + +Behavior: + +- The gateway fails to connect to your Argo CD instance +- The gateway pod logs contain: + +```text +transport: authentication handshake failed: EOF +``` + +Cause: + +- Your Argo CD instance is running without a certificate (e.g. SSL is terminated at a load balancer), while the gateway is configured by default to require encrypted traffic + +Resolution: + +- If it is intended that you don't have a certificate, you can disable encryption between the gateway and Argo CD by doing the following: + +```bash +helm upgrade --atomic \ +--version "1.0.0" \ +--namespace "{{GATEWAY_NAMESPACE}}" \ +--reset-then-reuse-values \ +--set gateway.argocd.insecure="false" \ +--set gateway.argocd.plaintext="true" \ +{{EXISTING_HELM_RELEASE_NAME}} \ +oci://registry-1.docker.io/octopusdeploy/octopus-argocd-gateway-chart +``` + +:::div{.warning} +By setting `gateway.argocd.plaintext="true"`, all traffic between the gateway and Argo CD will be unencrypted. Make sure this configuration is necessary to avoid potential security issues. +::: + ## Application/Project mapping ### No applications are listed on the **Argo CD Instance ➜ Applications** page From 1c19e02065d988f572cdc1846e5bd0daf417aa7e Mon Sep 17 00:00:00 2001 From: Denys Melnyk Date: Wed, 10 Jun 2026 18:49:57 +0300 Subject: [PATCH 2/3] Add troubleshooting section for gateway connectivity issues and update connection requirements --- src/pages/docs/argo-cd/instances/index.md | 12 ++++----- src/pages/docs/argo-cd/troubleshooting.md | 32 +++++++++++++++++++++++ 2 files changed, 38 insertions(+), 6 deletions(-) diff --git a/src/pages/docs/argo-cd/instances/index.md b/src/pages/docs/argo-cd/instances/index.md index 876b605091..ac615a9ce2 100644 --- a/src/pages/docs/argo-cd/instances/index.md +++ b/src/pages/docs/argo-cd/instances/index.md @@ -33,14 +33,14 @@ The [Kubernetes agent](/docs/kubernetes/targets/kubernetes-agent) and the [Kuber The gateway makes three outgoing connections. Before installing, make sure all of them are reachable from inside your cluster: -| The gateway must reach | How | -| --- | --- | -| Octopus Server REST API | HTTPS, port `443` | -| Octopus Server gRPC endpoint | gRPC (HTTP/2), port `8443` by default | -| Argo CD API server | In-cluster gRPC, e.g. `..svc.cluster.local` | +| Destination | Protocol | Port | +| --- | --- | --- | +| Octopus Server REST API | HTTPS | `443` | +| Octopus Server gRPC endpoint | gRPC (HTTP/2) | `8443` by default | +| Argo CD API server | gRPC (in-cluster) | Argo CD service port | :::div{.warning} -If your Octopus Server sits behind a load balancer, proxy, or firewall, make sure the gRPC port (`8443` by default) is forwarded to Octopus Server and the proxy supports HTTP/2. Forwarding only HTTPS (`443`) is a common cause of installation failure, where the gateway registers successfully but never connects. See [Troubleshooting](/docs/argo-cd/troubleshooting#argo-cd-gateway-registers-but-fails-to-connect-to-octopus-server) for details. +If your Octopus Server sits behind a load balancer, proxy, or firewall, make sure the gRPC port (`8443` by default) is forwarded to Octopus Server and the proxy supports HTTP/2. Forwarding only HTTPS (`443`) is a common cause of installation failure, where the gateway registers successfully but never connects. See [Troubleshooting](/docs/argo-cd/troubleshooting#failed-to-connect-to-octopus) for details. ::: :::div{.hint} diff --git a/src/pages/docs/argo-cd/troubleshooting.md b/src/pages/docs/argo-cd/troubleshooting.md index 8f3fffec1a..703e597695 100644 --- a/src/pages/docs/argo-cd/troubleshooting.md +++ b/src/pages/docs/argo-cd/troubleshooting.md @@ -164,6 +164,38 @@ oci://registry-1.docker.io/octopusdeploy/octopus-argocd-gateway-chart By setting `gateway.argocd.plaintext="true"`, all traffic between the gateway and Argo CD will be unencrypted. Make sure this configuration is necessary to avoid potential security issues. ::: +## Gateway Connectivity + +### Gateway connection drops at regular intervals (load balancer idle timeout) + +Behavior: + +- The gateway installs and connects successfully, but loses its connection to Octopus Server after every quiet period of the same length (e.g. 60 seconds without activity) +- Deployments with Argo CD steps fail intermittently with gRPC connection errors, and succeed when retried +- The "Gateway connectivity" tab of the Argo CD instance intermittently shows "Unavailable", depending on when the last health check ran +- The gateway pod logs show stream errors followed by an immediate reconnection +- If the load balancer drops connections silently instead of closing them, the logs show failing keep alives (`keep alive check failed - cancelling subscribers` with `DeadlineExceeded` errors) and the gateway pod restart count climbs at a regular cadence + +Cause: + +- A load balancer or proxy between the gateway and Octopus Server closes connections it considers idle +- The gateway sends a keep alive to Octopus Server every 30 seconds by default to hold the connection open. If the load balancer's idle timeout is shorter than the keep alive interval (or keep alives are disabled), the connection is terminated before the next keep alive is sent + +Resolution: + +- Increase the idle timeout on your load balancer so it comfortably exceeds the keep alive interval (`gateway.octopus.keepAlive.intervalSeconds`, default 30 seconds) +- Alternatively, reduce the keep alive interval below the load balancer's idle timeout: + +```bash +helm upgrade --atomic \ +--version "1.0.0" \ +--namespace "{{GATEWAY_NAMESPACE}}" \ +--reset-then-reuse-values \ +--set gateway.octopus.keepAlive.intervalSeconds="15" \ +{{EXISTING_HELM_RELEASE_NAME}} \ +oci://registry-1.docker.io/octopusdeploy/octopus-argocd-gateway-chart +``` + ## Application/Project mapping ### No applications are listed on the **Argo CD Instance ➜ Applications** page From 119d83fae3f3676b0653b141aca24928be1b4792 Mon Sep 17 00:00:00 2001 From: Denys Melnyk Date: Wed, 10 Jun 2026 19:17:36 +0300 Subject: [PATCH 3/3] Update modification date in documentation for Argo CD instances and troubleshooting sections --- src/pages/docs/argo-cd/instances/index.md | 2 +- src/pages/docs/argo-cd/troubleshooting.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/pages/docs/argo-cd/instances/index.md b/src/pages/docs/argo-cd/instances/index.md index ac615a9ce2..1071602445 100644 --- a/src/pages/docs/argo-cd/instances/index.md +++ b/src/pages/docs/argo-cd/instances/index.md @@ -1,7 +1,7 @@ --- layout: src/layouts/Default.astro pubDate: 2025-09-15 -modDate: 2026-06-10 +modDate: 2026-06-11 navSection: Argo CD Instances navTitle: Overview title: Overview diff --git a/src/pages/docs/argo-cd/troubleshooting.md b/src/pages/docs/argo-cd/troubleshooting.md index 703e597695..2688f7d566 100644 --- a/src/pages/docs/argo-cd/troubleshooting.md +++ b/src/pages/docs/argo-cd/troubleshooting.md @@ -1,7 +1,7 @@ --- layout: src/layouts/Default.astro pubDate: 2025-09-15 -modDate: 2026-06-10 +modDate: 2026-06-11 title: Troubleshooting Argo CD in Octopus navTitle: Troubleshooting description: How to resolve configuration issues