This action is brought to you by Snowflake, don't hesitate to visit our website and reach out to us if you have questions about this action.
When using the GitHub Flow Registry Client in NiFi to version control your flows, add the below file .github/workflows/flowdiff.yml to the repository into which flow definitions are versioned.
Whenever a pull request is opened, reopened or when a new commit is pushed to an existing pull request, this workflow will be triggered and will compare the modified flow(s) in the pull request and will automatically comment the pull request with a human readable description of the changes included in the pull request.
name: Snowflake Flow Diff on Pull Requests
on:
pull_request:
types: [opened, reopened, synchronize]
jobs:
execute_flow_diff:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
name: Executing Flow Diff
steps:
# checking out the code of the pull request (merge commit - if the PR is mergeable)
- name: Checkout PR code
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.ref }}
fetch-depth: 0
path: submitted-changes
# getting the path of the flow definition(s) that changed
- name: Get changed files
id: files
run: |
cd submitted-changes
# 1. grab every changed connector JSON
files=$(git diff --name-only $(git merge-base HEAD origin/${{ github.event.pull_request.base.ref }}) HEAD | grep '\.json$')
# 2. make it a comma-separated list (no trailing comma)
bare=$(echo "$files" | tr '\n' ',' | sed 's/,$//')
# 3. prefix for original-code (flowA)
flowA=$(echo "$bare" | sed 's|[^,]\+|original-code/&|g')
# 4. prefix for submitted-changes (flowB)
flowB=$(echo "$bare" | sed 's|[^,]\+|submitted-changes/&|g')
# 5. export both as outputs
echo "flowA=$flowA" >> $GITHUB_OUTPUT
echo "flowB=$flowB" >> $GITHUB_OUTPUT
# checking out the code without the change of the PR
- name: Checkout original code
uses: actions/checkout@v4
with:
fetch-depth: 2
path: original-code
- run: cd original-code && git checkout HEAD^
# Running the diff
- name: Snowflake Flow Diff
uses: snowflake-labs/snowflake-flow-diff@v0
id: flowdiff
with:
flowA: ${{ steps.files.outputs.flowA }}
flowB: ${{ steps.files.outputs.flowB }}Note - you may want to change grep '\.json$' with a more specific pattern to match your specific requirements.
This action works with GitHub Enterprise Server (GHE) out of the box. The GitHub API URL is automatically detected via github.api_url. If you need to override it for any reason, you can explicitly set the api-url input:
- name: Snowflake Flow Diff
uses: snowflake-labs/snowflake-flow-diff@v0
id: flowdiff
with:
flowA: ${{ steps.files.outputs.flowA }}
flowB: ${{ steps.files.outputs.flowB }}
api-url: https://github.example.com/api/v3Optionally, it is possible to enable a checkstyle check on the new version of the flow. If some violations against NiFi best practices are found, a message will be added to the comment published on the pull request.
To enable checkstyle:
# Running the diff
- name: Snowflake Flow Diff
uses: snowflake-labs/snowflake-flow-diff@v0
id: flowdiff
with:
flowA: ${{ steps.files.outputs.flowA }}
flowB: ${{ steps.files.outputs.flowB }}
checkstyle: true
# optional: path to YAML configuration of the rules
checkstyle-rules: submitted-changes/.github/checkstyle/checkstyle-rules.yaml
# optional: fail the action when violations are detected
checkstyle-fail: trueIf checkstyle-fail is set to true, the GitHub Action will exit with a non-zero status whenever checkstyle violations are detected, which ensures the workflow (and therefore the pull request) is blocked until the issues are fixed.
The YAML file can be used to include or exclude specific rules and to configure rule parameters. For example:
include:
- concurrentTasks
rules:
concurrentTasks:
parameters:
limit: 2
overrides:
".*sql-connector":
limit: 4
exclude:
- "Ignore.*"
componentExclusions:
"ProdFlow.*":
- "1a59f65f-8b3a-3db9-982e-e0d334bd7e9c" # processor UUID to ignoreAt the root level, if include is specified, only those rules will be executed. If exclude is specified AND include is not specified, all rules except the ones specified will be executed. For each rule, it is possible to specify default values for parameters (when supported by the rule, see below), to override parameters for specific flows using a regular expression against the flow name, to exclude flows using a regular expression, and to silence violations for specific component UUIDs via componentExclusions.
Some rules (for example concurrentTasks, noSelfLoop, enforcePrioritizer, or backpressureThreshold) may need an exception for a single processor or connection while you keep the rule enabled for the rest of the flow. You can scope exclusions down to the UUID under the componentExclusions map, keyed by flow-name regular expressions:
include:
- noSelfLoop
rules:
noSelfLoop:
componentExclusions:
"ProductionFlow.*":
- "2d8da922-fd1f-3519-9d54-6482dfd42c56"
".*": # optional global fallback
- "50a3b081-d54d-3ad8-b74c-caa7fef59bb2"When the flow name matches the regex, violations produced for the listed component IDs are ignored, while all other components continue to be checked normally.
Use the same pattern for rules that operate on connections (for example enforcePrioritizer or backpressureThreshold) by listing the connection identifiers to suppress.
The following table summarizes what each rule reports and which identifiers you can expect when configuring componentExclusions:
| Rule id | Violation references | componentExclusions target |
|---|---|---|
concurrentTasks |
Processor name and UUID | Processor UUID (VersionedProcessor#getIdentifier) |
snapshotMetadata |
Flow name only | Not applicable |
emptyParameter |
Parameter name | Not applicable |
defaultParameters |
Parameter name | Not applicable |
unusedParameter |
Parameter name | Not applicable |
noSelfLoop |
Component name/type and UUID | Processor/Funnel UUID at both ends of the self-loop |
enforcePrioritizer |
Connection description and UUID | Connection UUID (VersionedConnection#getIdentifier) |
backpressureThreshold |
Connection description and UUID | Connection UUID (VersionedConnection#getIdentifier) |
processorNaming |
Processor name, type, and UUID | Processor UUID (VersionedProcessor#getIdentifier) |
controllerServiceNaming |
Controller Service name, type, and UUID | Controller Service UUID (VersionedControllerService#getIdentifier) |
parameterContextNaming |
Parameter Context name | Not applicable |
parameterProviderNaming |
Parameter Provider name, type, and UUID | Not applicable |
Available rules:
concurrentTasksto check the number of concurrent tasks and define an upper limit (parameter:limit, default value is 2)snapshotMetadatato check that the flow snapshot metadata is present in the fileemptyParameterto check that no parameter is set to an empty stringdefaultParametersto check parameters with a default value (parameter:defaultParameters, comma-separated list of parameter names having an expected default value)unusedParameterto check if all specified parameters are used in the flownoSelfLoopto check if there are self-loop connections in the flowenforcePrioritizerto check if all connections in the flow are set with the configured list of prioritizers (parameter:prioritizers, comma-separated list of expected prioritizers, example:org.apache.nifi.prioritizer.FirstInFirstOutPrioritizer)backpressureThresholdto ensure each connection keeps both data size and object count backpressure thresholds greater than zeroprocessorNamingto validate processor names against regex patterns based on processor type (parameters:patternsmap of fully qualified type to regex, optionaldefaultPatternfor types not in the map). This rule produces no violations when no patterns are configured.controllerServiceNamingto validate controller service names against regex patterns based on service type (parameters:patternsmap of fully qualified type to regex, optionaldefaultPatternfor types not in the map). This rule produces no violations when no patterns are configured.parameterContextNamingto validate parameter context names against a regex pattern (parameters:defaultPatternregex, optionalexcludelist of exact names to skip). This rule produces no violations when no pattern is configured.parameterProviderNamingto validate parameter provider names against regex patterns based on provider type (parameters:patternsmap of fully qualified type to regex, optionaldefaultPatternfor types not in the map). This rule produces no violations when no patterns are configured.
The naming rules (processorNaming, controllerServiceNaming, parameterContextNaming, parameterProviderNaming) allow you to enforce custom naming conventions on NiFi components. They are no-ops without configuration, so they are safe to include without side effects.
Patterns are evaluated with Java's String.matches(), which requires the entire name to match (as if the pattern were implicitly anchored with ^ and $). For example, to match names ending in _SUFFIX, use .*_SUFFIX$ rather than _SUFFIX$.
Validate processor names by type. Use patterns to map fully qualified processor types to regex patterns, and defaultPattern as a fallback for types not explicitly listed:
include:
- processorNaming
rules:
processorNaming:
parameters:
patterns:
org.apache.nifi.processors.snowflake.PutSnowflakeInternalStage: ".*_ISTG$"
org.apache.nifi.processors.standard.ExecuteSQLRecord: ".*_(SEL|COPY|UPD_SEL|UPD_INS|EXT)$|^SP_.*"
org.apache.nifi.processors.standard.GenerateFlowFile: "^GENERATE_FLOW_FILE$"
org.apache.nifi.processors.standard.UpdateAttribute: "^UPDATE_ATTRIBUTE$"
org.apache.nifi.processors.standard.InvokeHTTP: ".*_EXT$"
defaultPattern: "^[A-Z][A-Z0-9_]+$"
componentExclusions:
"ProductionFlow.*":
- "some-processor-uuid-to-ignore"Validate controller service names by type. Works the same as processor naming:
include:
- controllerServiceNaming
rules:
controllerServiceNaming:
parameters:
patterns:
org.apache.nifi.dbcp.DBCPConnectionPool: "^acme_(dev|preprod|prod)_[a-z0-9_]+$"
defaultPattern: "^[a-z][a-z0-9_]+$"Validate parameter context names against a single pattern. Use exclude to skip specific context names:
include:
- parameterContextNaming
rules:
parameterContextNaming:
parameters:
defaultPattern: "^acme_(dev|preprod|prod)_[a-z0-9_]+_(secrets|config)$"
exclude:
- "common_parameter_context"
- "common_sftp_serverkey"
- "CommonDatabaseDriversReference"Validate parameter provider (management controller service) names by type:
include:
- parameterProviderNaming
rules:
parameterProviderNaming:
parameters:
patterns:
org.apache.nifi.parameter.aws.AwsSecretsManagerParameterProvider: "^acme_(dev|preprod|prod)_[a-z0-9_]+_secrets$"All naming rules support the standard overrides mechanism to apply different patterns per flow name, and componentExclusions to silence violations for specific component UUIDs (where applicable). When a per-flow override specifies patterns, it replaces the entire top-level patterns map for matching flows rather than merging with it.
Before computing a diff, the action validates both flow files for duplicate JSON keys. Duplicate keys are not valid JSON and silently produce wrong diffs because parsers apply last-wins on repeated keys, discarding earlier occurrences. This commonly occurs when a merge conflict is not fully resolved.
If a duplicate key is detected the action posts a [!CAUTION] block in the PR comment identifying the file and the exact line, then exits with a non-zero status code so the PR check is blocked until the file is fixed:
> [!CAUTION]
> Flow file `submitted-changes/flows/my-flow.json` contains duplicate JSON keys (this typically indicates a merge conflict that was not fully resolved): Duplicate field 'flowContents'
> Line 37, column 17The GitHub Action will automatically publish a comment on the pull request with a comprehensive description of the changes between the flows of the two branches. Here is an example of what the comment could look like:
### Executing Snowflake Flow Diff for flow: `MyExample`
#### Checkstyle Violations
> [!CAUTION]
> - Processor named `UpdateAttribute` is configured with 4 concurrent tasks
#### Flow Changes
**`MyExample`** — 8 changes
- The destination of a connection has changed from `UpdateAttribute` to `InvokeHTTP`
- A self-loop connection `[success]` has been added on `UpdateAttribute`
- A connection `[success]` from `My Generate FlowFile Processor` to `UpdateAttribute` has been added
- A Processor of type `GenerateFlowFile` named `GenerateFlowFile` has been renamed from `GenerateFlowFile` to `My Generate FlowFile Processor`
- In Processor of type `GenerateFlowFile` named `GenerateFlowFile`, the Scheduling Strategy changed from `TIMER_DRIVEN` to `CRON_DRIVEN`
- The Parameter Context `Test Parameter Context` with parameters `{addedParam=newValue}` has been added to the process group `TestingFlowDiff`
- A Processor of type `UpdateAttribute` named `UpdateAttribute` has been removed
- In Processor of type `GenerateFlowFile` named `GenerateFlowFile`, the Run Schedule changed from `1 min` to `* * * * * ?`
**`MyExample > Ingestion`** — 1 change
- A Processor of type `UpdateAttribute` named `UpdateAttribute` has been added with the configuration [`ALL` nodes, `4` concurrent tasks, `0ms` run duration, `WARN` bulletin level, `TIMER_DRIVEN` (`0 sec`), `30 sec` penalty duration, `1 sec` yield duration] and the below properties:
- `Store State` = `Do not store state`
- `canonical-value-lookup-cache-size` = `100`
**`(Parameter Contexts)`** — 2 changes
- A Parameter Context named `Test Parameter Context` has been added
- A Parameter Context named `Another one to delete` has been added
#### Bundle Changes
- The bundle `org.apache.nifi:nifi-standard-nar` has been changed from version `2.1.0` to version `2.2.0`