Multi Cluster join work in UI but fails when declarative

[jDmacD]

Joining via UI

This works as expected.

Joining using helm values

This results in Request error - error=Invalid data

Master rke2-a-utilty

internal:
  certmanager:
    enabled: true
    secretname: neuvector-internal

enforcer:
  internal:
    certificate:
      secret: neuvector-internal

cve:
  scanner:
    internal:
      certificate:
        secret: neuvector-internal

manager:
  env:
    ssl: false
  ingress:
    enabled: true
    host: neuvector.xxxxxx.io
    ingressClassName: cilium
    path: /
    annotations:
      external-dns.alpha.kubernetes.io/hostname: neuvector.xxxxxx.io
      cert-manager.io/cluster-issuer: letsencrypt-production-issuer
    tls: true
    secretName: ui-letsencrypt-production-certificate

controller:
  replicas: 1
  env:
    - name: CTRL_PATH_DEBUG
      value: debug
  federation:
    mastersvc:
      type: ClusterIP
      ingress:
        enabled: true
        host: neuvector.utility.xxxxxx.io
        ingressClassName: cilium
        path: /
        annotations:
          external-dns.alpha.kubernetes.io/hostname: neuvector.utility.xxxxxx.io
          ingress.cilium.io/tls-passthrough: enabled
        tls: false
  internal:
    certificate:
      secret: neuvector-internal

  configmap:
    enabled: true
    data:
      # passwordprofileinitcfg.yaml: |
      #  ...
      # roleinitcfg.yaml: |
      #  ...
      # ldapinitcfg.yaml: |
      #  ...
      # oidcinitcfg.yaml: |
      # ...
      # samlinitcfg.yaml: |
      # ...
      # sysinitcfg.yaml: |
      # ...
      # userinitcfg.yaml: |
      # ...
      fedinitcfg.yaml: |
        always_reload: true
        Cluster_Name: rke2-a-utility
        Join_Token: 8775feda-657f-4fc7-8958-4be3e7e13648
        Primary_Rest_Info:
          Server: neuvector.utility.xxxxxx.io
          Port: 443
        Deploy_Repo_Scan_Data: true
        Use_Proxy: ""

2025-08-23T12:26:04.295|DEBU|CTL|rest.handlerJoinFedInternal: - URL=/v1/fed/join_internal
2025-08-23T12:26:04.302|DEBU|CTL|cluster.Put: - key=state/dist_lock/federation value={"locked_by":"7d33c849d1a6cbaf8e97912b036faa4062f03d07aa9dac609b668950f4d2bea9","locked_at":"2025-08-23T12:26:04.302277349Z","caller":"rest.handlerJoinFedInternal"}
2025-08-23T12:26:04.305|DEBU|CTL|rest.handlerJoinFedInternal: a new joint cluster wants to join
2025-08-23T12:26:04.306|ERRO|CTL|rest.handlerJoinFedInternal: Request error - error=Invalid data
2025-08-23T12:26:04.306|DEBU|CTL|rest.writer.WriteHeader: 417 - Method=POST URL=/v1/fed/join_internal
2025-08-23T12:26:04.306|DEBU|CTL|cluster.Delete: - key=state/dist_lock/federation

Managed rke2-d-apps

internal:
  certmanager:
    enabled: true
    secretname: neuvector-internal

enforcer:
  internal:
    certificate:
      secret: neuvector-internal

cve:
  scanner:
    internal:
      certificate:
        secret: neuvector-internal

manager:
  enabled: true

controller:
  replicas: 1
  env:
    - name: CTRL_PATH_DEBUG
      value: debug
  federation:
    managedsvc:
      type: ClusterIP
      ingress:
        enabled: true
        host: neuvector.apps.xxxxxx.io
        ingressClassName: cilium
        path: /
        annotations:
          external-dns.alpha.kubernetes.io/hostname: neuvector.apps.xxxxxx.io
          ingress.cilium.io/tls-passthrough: enabled
        tls: false
  internal:
    certificate:
      secret: neuvector-internal

  configmap:
    enabled: true
    data:
      # passwordprofileinitcfg.yaml: |
      #  ...
      # roleinitcfg.yaml: |
      #  ...
      # ldapinitcfg.yaml: |
      #  ...
      # oidcinitcfg.yaml: |
      # ...
      # samlinitcfg.yaml: |
      # ...
      # sysinitcfg.yaml: |
      # ...
      # userinitcfg.yaml: |
      # ...
      fedinitcfg.yaml: |
        always_reload: true
        Cluster_Name: rke2-d-apps
        Join_Token: 8775feda-657f-4fc7-8958-4be3e7e13648
        Primary_Rest_Info:
          Server: neuvector.utility.xxxxxx.io
          Port: 443
        Managed_Rest_Info:
          Server: neuvector.apps.xxxxxx.io
          Port: 443
        Use_Proxy: ""

2025-08-23T12:26:04.312|ERRO|CTL|rest.sendRestReqInternal: Request failed - proxyOption=0 status=417 Expectation Failed timeout=10s url=https://neuvector.utility.heanet.io:443/v1/fed/join_internal
2025-08-23T12:26:04.314|ERRO|CTL|rest.joinFed: - data={"code":39,"error":"Federation operation failed","message":"Invalid data"} kv_version=94444768 localServer={Server:neuvector.apps.xxxxxx.io Port:443} masterServer={Server:neuvector.utility.xxxxxx.io Port:443} proxyUsed=false statusCode=417
2025-08-23T12:26:04.314|DEBU|CTL|rest.handlefedcfg: join - err=Federation operation failed

[DanDens]

Hi,

can verify that I see the same issue. Made sure all CRDs and Containers are the same. Still see the issue with the configmap, but joining via UI is working fine.

[williamlin-suse]

It means the Join_Token in fedinitcfg.yaml on primary cluster is not loaded yet when the managed cluster tries to join it using the Join_Token in fedinitcfg.yaml on the managed cluster.
A full controller logs on primary/manager clusters do help for the investigation