DOC-2235: Document the Visual pipeline editor for Redpanda Connect

.github/workflows/doc-detective-screenshots.yml

@@ -0,0 +1,125 @@
+name: Cloud console screenshot tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      environment:
+        description: Which environment to test
+        type: choice
+        options: [preprod, prod, both]
+        default: both
+  schedule:
+    # Daily at 13:00 UTC. Catches upcoming Cloud console UI changes in preprod
+    # before they ship to production, and keeps docs screenshots in sync.
+    - cron: '0 13 * * *'
+
+permissions:
+  contents: read
+
+jobs:
+  set-matrix:
+    runs-on: ubuntu-latest
+    outputs:
+      environments: ${{ steps.set.outputs.environments }}
+    steps:
+      - id: set
+        run: |
+          choice="${{ inputs.environment }}"
+          if [ -z "$choice" ] || [ "$choice" = "both" ]; then
+            echo 'environments=["preprod","prod"]' >> "$GITHUB_OUTPUT"
+          else
+            echo "environments=[\"$choice\"]" >> "$GITHUB_OUTPUT"
+          fi
+
+  screenshot-test:
+    needs: set-matrix
+    name: ${{ matrix.environment }}
+    runs-on: ubuntu-latest
+    # doc-detective/github-action needs write access to open the PR (screenshot
+    # changed) and/or the issue (a step failed) described below — broader than
+    # the workflow-level `contents: read` default, scoped to just this job.
+    permissions:
+      contents: write
+      pull-requests: write
+      issues: write
+    strategy:
+      fail-fast: false
+      matrix:
+        environment: ${{ fromJson(needs.set-matrix.outputs.environments) }}
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+
+      - name: Create .env file for this environment
+        run: |
+          if [ "${{ matrix.environment }}" = "preprod" ]; then
+            echo "CLOUD_BASE_URL=https://preprod--redpanda-cloud.netlify.app" >> tests/doc-detective/.env
+            echo "CLOUD_LOGIN_EMAIL=${{ secrets.PREPROD_EMAIL }}" >> tests/doc-detective/.env
+            echo "CLOUD_LOGIN_PASSWORD=${{ secrets.PREPROD_PASSWORD }}" >> tests/doc-detective/.env
+            echo "CLOUD_CLUSTER_ID=d9f1nf0d6v8fd1orsa00" >> tests/doc-detective/.env
+          else
+            echo "CLOUD_BASE_URL=https://cloud.redpanda.com" >> tests/doc-detective/.env
+            echo "CLOUD_LOGIN_EMAIL=${{ secrets.EMAIL }}" >> tests/doc-detective/.env
+            echo "CLOUD_LOGIN_PASSWORD=${{ secrets.PASSWORD }}" >> tests/doc-detective/.env
+            echo "CLOUD_CLUSTER_ID=cnk4fdktluj4mbfiarq0" >> tests/doc-detective/.env
+          fi
+
+      # Runs the same tests/doc-detective/config.json used locally (see
+      # run-inline.js) via the official Doc Detective action instead of our
+      # own wrapper script, so drift/outdated-screenshot detection and
+      # failure reporting are handled by tooling doc-detective itself
+      # maintains, not custom glue here.
+      #
+      # exit_on_fail + create_issue_on_fail: a genuine test failure (e.g. a
+      # selector changed) both fails this job (visible at a glance in the
+      # Actions tab) and files an issue. On prod specifically, that's
+      # expected to fire until the canvas feature flag ships there — that's
+      # a real, if noisy, signal that the flag hasn't flipped yet.
+      #
+      # create_pr_on_change: if the run made it all the way to the
+      # screenshot step and the real canvas no longer matches the checked-in
+      # image closely enough (see maxVariation in the inline test), that
+      # step overwrites the file and this opens a PR with the diff. A
+      # reviewer still has to look at the picture before merging — this
+      # surfaces drift for review, it doesn't silently accept it.
+      #
+      # Known residual risk, not yet exercised end-to-end: this action
+      # installs its own copy of doc-detective rather than using this repo's
+      # node_modules, so it doesn't get the sharp/@img version-pinning fix in
+      # this repo's package.json overrides (see tests/doc-detective/README.md
+      # and https://github.com/appium/appium/issues/22554). If a fresh
+      # install of the action's own dependencies hits that same upstream
+      # conflict, the screenshot step could fail with a native module error
+      # unrelated to the actual docs/UI. Validate with a manual
+      # workflow_dispatch run before trusting the daily cron on this.
+      - name: Run Doc Detective
+        uses: doc-detective/github-action@v1
+        with:
+          config: tests/doc-detective/config.json
+          exit_on_fail: true
+          create_pr_on_change: true
+          pr_branch: doc-detective-${{ matrix.environment }}-${{ github.run_id }}
+          pr_title: "docs: update ${{ matrix.environment }} Visual editor screenshot"
+          pr_body: |
+            A scheduled Doc Detective run against **${{ matrix.environment }}** ($RUN_URL) found the real Visual pipeline editor canvas no longer matches the checked-in screenshot closely enough, and updated it.
+
+            This only means pixels changed by more than the accepted threshold — review the image diff before merging, same as any other screenshot change.
+          pr_labels: doc-detective,${{ matrix.environment }}
+          create_issue_on_fail: true
+          issue_title: "Doc Detective failure: ${{ matrix.environment }} Visual editor test"
+          # Deliberately does NOT use the $RESULTS template variable: the raw
+          # results object embeds resolved step values, including the
+          # CLOUD_LOGIN_PASSWORD/CLOUD_LOGIN_EMAIL substituted into this
+          # test's login steps. That object goes into the issue body via a
+          # direct API call, bypassing GitHub's log secret-masking (which
+          # only scrubs step *logs*, not content posted through the API) —
+          # so putting it here would leak the password in plaintext into the
+          # issue. $RUN_URL points at the (correctly masked) run logs instead.
+          issue_body: |
+            A Doc Detective run against **${{ matrix.environment }}** failed: $RUN_URL
+
+            The Visual pipeline editor test in `connect-quickstart.adoc` no longer matches the real UI. Check the run logs for which step failed and why — most likely a selector changed, or (if this is the `prod` run) the canvas feature flag hasn't shipped there yet.
+          issue_labels: doc-detective,${{ matrix.environment }}

.github/workflows/screenshot-standards-lint.yml

@@ -0,0 +1,31 @@
+name: Screenshot standards lint
+
+on:
+  pull_request:
+    branches:
+      - main
+    paths:
+      - '**/*.adoc'
+      - 'modules/shared/images/**'
+
+permissions:
+  contents: read
+
+jobs:
+  lint:
+    name: Check alt text and image size
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+
+      - name: Lint screenshots and alt text
+        run: node tests/doc-detective/lint-screenshots.js

.gitignore

@@ -9,3 +9,5 @@ poc-docs/
 docs/
 .docusaurus
 .env
+tests/doc-detective/last-report.json
+tests/doc-detective/last-inline-report.json

modules/develop/pages/connect/connect-quickstart.adoc

@@ -77,7 +77,7 @@ The *consumer pipeline* uses the following Redpanda Connect components:
 |Catches errors 
 |===  
 
-TIP: The pipeline editor provides an IDE-like experience for creating pipelines. After a component has been added, you can click the leaf icon in the left sidebar to open its documentation.
+TIP: The pipeline editor has two views: *YAML* and *Visual*. The *YAML* view provides an IDE-like experience for creating pipelines: after a component has been added, you can click the leaf icon in the left sidebar to open its documentation. The *Visual* view renders your pipeline as an interactive node diagram so you can inspect and edit each component without hand-writing YAML. Both views stay in sync, so you can switch between them at any time. This quickstart uses the *YAML* view. To try the *Visual* view instead, see xref:develop:connect/connect-quickstart.adoc#visual-editor[Explore the Visual editor].
 
 image::shared:connect_ui.png[Redpanda Connect user interface]
 
@@ -390,9 +390,101 @@ When you've finished experimenting with your data pipeline, you can delete the p
 . Confirm your deletion to remove the pipelines and associated logs.
 . On the *Topics* page, delete the `dad-jokes` topic.
 
+[#visual-editor]
+== Explore the Visual editor
+// (test {"testId":"visual-editor-quickstart","description":"Verifies the procedure in this section still matches the real UI: switching tabs, opening the node inspector, the / search palette, the Problems/Unsaved indicators, the Save button, and the YAML-tab structure tree. Then builds a representative pipeline via the YAML tab and captures it as the canvas screenshot referenced above. One test, one login, one browser session for both, since separate tests each get their own fresh, unauthenticated browser session. Browser context, pinned Chrome at 1280x800, comes from runOn in run-inline.js."})
+// (step {"loadVariables":"tests/doc-detective/.env"})
+// (step {"goTo":"$CLOUD_BASE_URL"})
+// (step {"find":{"selector":"input[type=email]","timeout":20000}})
+// (step {"type":{"keys":["$CLOUD_LOGIN_EMAIL","$ENTER$"],"selector":"input[type=email]"}})
+// (step {"wait":2000})
+// (step {"find":{"selector":"input[type=password], [data-testid='connection-Username-Password-Authentication-button']","click":true,"timeout":10000}})
+// (step {"wait":2000})
+// (step {"find":{"selector":"#password","click":true,"timeout":10000}})
+// (step {"type":{"keys":["$CLOUD_LOGIN_PASSWORD","$ENTER$"]}})
+// (step {"wait":3000})
+// (step {"goTo":"$CLOUD_BASE_URL/clusters/$CLOUD_CLUSTER_ID/rp-connect/create"})
+// (step {"wait":3000})
+// (step {"goTo":"$CLOUD_BASE_URL/clusters/$CLOUD_CLUSTER_ID/rp-connect/create"})
+// (step {"wait":5000})
+
+Instead of writing YAML, you can build and edit a pipeline on the *Visual* tab, an interactive node diagram of your pipeline's inputs, processors, outputs, and control-flow constructs like `switch` and `try`/`catch`.
+
+image::shared:rpcn-visual-editor-canvas-v26-2.png[The Visual tab showing a generate input node on the pipeline canvas]
+
+. Open an existing pipeline, or create a new one, and click the *Visual* tab (next to *YAML*).
+// (step {"find":{"selector":"[data-value='yaml']","click":true,"timeout":15000}})
+// (step {"wait":500})
+// (step {"find":{"selector":"[data-value='visual']","click":true,"timeout":10000}})
+// (step {"wait":500})
+// (step {"find":{"selector":".react-flow","timeout":10000}})
+// (step {"find":{"elementText":"Add input","click":true,"timeout":15000}})
+// (step {"wait":500})
+// (step {"type":{"keys":["generate"],"selector":"input[cmdk-input]"}})
+// (step {"wait":500})
+// (step {"find":{"elementText":"Add to pipeline","click":true,"timeout":5000}})
+// (step {"wait":1500})
+. Click a node on the canvas to inspect or edit its configuration in the side panel. Fields are generated from the component's schema. Components without a generated form, such as raw Bloblang mappings, show an editable YAML snippet instead.
+// (step {"find":{"selector":"[data-testid='rf__node-input-0']","click":true,"timeout":10000}})
+// (step {"wait":500})
+// (step {"find":{"selector":"aside","elementText":"/generate/","timeout":5000}})
+// (step {"type":{"keys":["$ESCAPE$"]}})
+// (step {"wait":300})
+. To add a component, click a plus-sign icon on an empty input or output slot, or on a connector line to insert a step in the middle of the pipeline. You can also press kbd:[/] to search for a connector by name. Canvas nodes aren't draggable; use these plus-sign icons and the search dialog to build the pipeline instead of dragging nodes into place.
+// (step {"type":{"keys":["/"]}})
+// (step {"find":{"selector":"[role='dialog']","timeout":5000}})
+// (step {"type":{"keys":["$ESCAPE$"]}})
+// (step {"wait":300})
+. Watch for two floating indicators on the canvas while you work:
+* *Problems* lists lint errors and missing secrets. Click an entry to jump to the node that needs attention.
+// (step {"find":{"selector":"[data-testid='pipeline-problems-chip']","timeout":5000}})
+* *Unsaved* lists nodes you've changed since the last save.
+// (step {"find":{"selector":"[data-testid='pipeline-unsaved-chip']","timeout":5000}})
+. Click *Save*. Edits you made on the *Visual* tab are written to the pipeline's YAML at this point.
+// (step {"find":{"elementText":"Save","click":true,"timeout":5000}})
+
+TIP: On the *YAML* tab, a structure tree replaces the sidebar's connector shortcuts once your pipeline has at least one node, letting you jump to any node in the outline.
+// (step {"find":{"selector":"[data-value='yaml']","click":true,"timeout":10000}})
+// (step {"wait":1000})
+// (step {"find":{"selector":"[role='tree']","timeout":5000}})
+// (step {"goTo":"$CLOUD_BASE_URL/clusters/$CLOUD_CLUSTER_ID/rp-connect/create"})
+// (step {"wait":3000})
+// (step {"goTo":"$CLOUD_BASE_URL/clusters/$CLOUD_CLUSTER_ID/rp-connect/create"})
+// (step {"wait":5000})
+// (step {"find":{"selector":"[data-value='yaml']","click":true,"timeout":15000}})
+// (step {"wait":1000})
+// (step {"find":{"selector":".monaco-editor textarea.inputarea","click":true,"timeout":10000}})
+// (step {"type":{"keys":["$CONTROL$","a"]}})
+// (step {"type":{"keys":["input:","$ENTER$"]}})
+// (step {"type":{"keys":["generate:","$ENTER$"]}})
+// (step {"type":{"keys":["interval: 1s","$ENTER$"]}})
+// (step {"type":{"keys":["count: 0","$ENTER$"]}})
+// (step {"type":{"keys":["mapping: 'root = \"Generated a message\"'","$ENTER$"]}})
+// (step {"type":{"keys":["$SHIFT$",""]}})
+// (step {"wait":300})
+// (step {"type":{"keys":["ppipeline:","$ENTER$"]}})
+// (step {"type":{"keys":["processors:","$ENTER$"]}})
+// (step {"type":{"keys":["- log:","$ENTER$"]}})
+// (step {"type":{"keys":["level: INFO","$ENTER$"]}})
+// (step {"type":{"keys":["message: \"Generated a message\"","$ENTER$"]}})
+// (step {"type":{"keys":["$SHIFT$",""]}})
+// (step {"wait":300})
+// (step {"type":{"keys":["ooutput:","$ENTER$"]}})
+// (step {"type":{"keys":["drop: {}"]}})
+// (step {"wait":500})
+// (step {"find":{"selector":"[data-value='visual']","click":true,"timeout":10000}})
+// (step {"wait":3000})
+// (step {"find":{"selector":".react-flow","timeout":10000}})
+// (step {"find":{"selector":"[aria-label='Fit View']","click":true,"timeout":5000}})
+// (step {"wait":1000})
+// (step {"screenshot":{"path":"modules/shared/images/rpcn-visual-editor-canvas-v26-2.png","crop":{"selector":".react-flow","padding":{"top":0,"right":16,"bottom":0,"left":16}},"overwrite":"aboveVariation","maxVariation":0.05}})
+// (test end)
+
+Edits in the *Visual* tab don't require an explicit apply step for each node: making changes to a node's configuration and then closing its panel or selecting a different node writes your edit into the YAML immediately, without needing to click *Save* first. If the YAML becomes invalid while you're editing (for example, from a syntax error made in the *YAML* tab), the *Visual* tab can't render it and shows *Unable to visualize this pipeline* until the YAML is valid again.
+
 == Next steps
 
-* Try one of the xref:cookbooks:index.adoc[Redpanda Connect cookbooks]. 
+* Try one of the xref:cookbooks:index.adoc[Redpanda Connect cookbooks].
 * Choose xref:develop:connect/components/about.adoc[connectors for your use case].
 * xref:develop:connect/configuration/secret-management.adoc[Add secrets to your pipeline].
 * xref:develop:connect/configuration/monitor-connect.adoc[Monitor a data pipeline on a BYOC or Dedicated cluster].

modules/get-started/pages/whats-new-cloud.adoc

@@ -33,6 +33,7 @@ You can now permanently delete a Serverless organization (free trial and pay-as-
 
 === Redpanda Connect updates
 
+* The pipeline editor has a new *Visual* tab alongside the existing *YAML* editor. The *Visual* tab renders a pipeline as an interactive node diagram, so you can inspect and edit inputs, processors, outputs, and control-flow constructs (`switch`, `branch`, `try`/`catch`, and others) without hand-writing YAML. The two views stay in sync. See xref:develop:connect/connect-quickstart.adoc#visual-editor[Explore the Visual editor].
 * Inputs:
 ** xref:develop:connect/components/inputs/jira.adoc[jira]: Streams Jira issues, comments, or changelog entries via JQL with incremental polling.