Support incremental_strategy parameter and new insert_overwrite strategy by SuchodolskiEdvin · Pull Request #2195 · dataform-co/dataform

SuchodolskiEdvin · 2026-06-02T14:39:33Z

updated proto with new parameters
added new tests
added validation for chosen incremental_strategies
added new insert_overwrite strategy logic

kolina · 2026-06-07T20:11:14Z

+    const backtickedColumns = columns.map(column => `\`${column}\``);
+    const resolveTargetTable = this.resolveTarget(target);
+
+    return `CREATE OR REPLACE TEMP TABLE \`${stagingTableUnqualified}\` AS (


can you split separate SQL statements into granular Task.statement calls?

I believe we can't split this SQL because we are using TEMP table. We could in case we would use persistent table.

I think what Nick means here is to split the sql creation into statements not the sql itself.

@kolina can you clarify what exactly do you mean?

kolina · 2026-06-07T20:30:48Z

+  MERGE ${resolveTargetTable} T
+  USING \`${stagingTableUnqualified}\` S
+  ON FALSE
+  WHEN NOT MATCHED BY SOURCE AND ${partitionBy} IN UNNEST(partitions_for_replacement) ${updatePartitionFilter ? `and T.${updatePartitionFilter}` : ""} THEN


Such code T.${updatePartitionFilter} will only work when updatePartitionFilter has exactly one expression?

Yes, you are correct. It is a known limitation that T.${updatePartitionFilter} only works for simple expressions (and fails on multi-expression SQL). Current implementation is designed to match the existing behavior of the standard MERGE strategy to maintain consistency between the two strategies for now. The fix of using updatePartitionFilter with several expression will be introduced in a separate PR.

kolina · 2026-06-07T20:35:18Z

+        return this.mergeInto(
+          table.target,
+          columns,
+          incrementalQuery,


Can you deduplicate this with a call below?

kolina · 2026-06-07T20:36:47Z

+    switch (this.proto.incrementalStrategy) {
+      case dataform.IncrementalStrategy.INSERT_OVERWRITE:
+        if (!this.proto.bigquery || !this.proto.bigquery.partitionBy) {
+          this.session.compileError(
+            new Error("incrementalStrategy 'insert_overwrite' requires 'partitionBy' to be set."),
+            config.filename,
+            this.proto.target
+          );
+        }
+        break;
+      case dataform.IncrementalStrategy.MERGE:
+        if (!this.proto.uniqueKey || this.proto.uniqueKey.length === 0) {
+          this.session.compileError(
+            new Error("incrementalStrategy 'merge' requires 'uniqueKey' to be set."),
+            config.filename,
+            this.proto.target
+          );
+        }
+        break;
+      default:
+        break;
+    }


let's move it into sub-function for readability

kolina · 2026-06-07T20:45:31Z

+      expect(result.compile.compiledGraph.tables[0].incrementalStrategy).equals(
+        dataform.IncrementalStrategy.INSERT_OVERWRITE
+      );


let's check the whole object here (same below)

kolina · 2026-06-07T20:49:10Z

    string reservation = 27;
+
+    // Optional. The incremental strategy to use when updating the table.
+    // Defaults to MERGE if uniqueKey is configured, or APPEND otherwise.


I wouldn't include it into config protos (because it's run-time strategy and not related to compiled DAG generation)

Tuseeq1 · 2026-06-12T08:41:03Z

+
+      const result = runMainInVm(coreExecutionRequestFromPath(projectDir));
+
+      expect(result.compile.compiledGraph.graphErrors.compilationErrors.length).greaterThan(0);


I think this should be checked to be exactly equal to 1 to avoid any other uncaught errors the code change might cause.

Tuseeq1 · 2026-06-12T08:41:56Z

+
+      const result = runMainInVm(coreExecutionRequestFromPath(projectDir));
+
+      expect(result.compile.compiledGraph.graphErrors.compilationErrors.length).greaterThan(0);


Tuseeq1 · 2026-06-12T08:44:53Z

+      case dataform.IncrementalStrategy.INSERT_OVERWRITE:
+        if (!this.proto.bigquery || !this.proto.bigquery.partitionBy) {
+          this.session.compileError(
+            new Error("incrementalStrategy 'insert_overwrite' requires 'partitionBy' to be set."),


Nit: Inconsistent style. in some places we have capital first letter and in others we don't. We also use "." in some places and miss it in others. for example on line 759.

Tuseeq1 · 2026-06-12T09:06:51Z

+    target: dataform.ITarget,
+    columns: string[],
+    query: string,
+    partitionBy: string,


What happens if the table.bigquery or table.bigquery.partitionBy is not provided? will it be an empty string or 'null'?

If table.bigquery or table.bigquery.partitionBy is not provided or it is empty string - insertOverwrite() won't be reached. There is validation of partitionBy parameter.

the validation link doesnt lead anywhere so I cannot check that.

It's strange, because it's working for me. This is link to the code change in scope of this PR. Please check core/actions/incremental_table.ts:checkIncrementalStrategyRequirements() (LINE 778)

Tuseeq1 · 2026-06-12T09:09:29Z

+    ARRAY(
+      SELECT DISTINCT ${partitionBy}
+      FROM \`${stagingTableUnqualified}\`
+      WHERE ${partitionBy} IS NOT NULL


What will this statement look like if we have empty partitionBy? also will it have `` around the column name somehow? Do we have an example for final sql that will be generated.

If partitionBy would contain column wrapped in function (like Date(ts)) adding backticks cause BigQuery to treat the entire expression as a literal column name.

Yes, we have golden files covering the generated SQL for insertOverwrite in cli/api/goldens/insert_overwrite*.sql

this is question I have. So if PartitionBy is not a function like Date and is just a column name lets say id. This will not add backticks to it. Will the query run in that case.

Yes, the query will run successfully, assuming the column isn't a reserved BigQuery keyword (like order). If it is a keyword, the user can still provide the column name wrapped in backticks directly in their config. Just FYI: this matches the existing behavior of the MERGE strategy, which also injects uniqueKey columns without automatically adding backticks.

Tuseeq1 · 2026-06-12T09:15:16Z

+    const backtickedColumns = columns.map(column => `\`${column}\``);
+    const resolveTargetTable = this.resolveTarget(target);
+
+    return `CREATE OR REPLACE TEMP TABLE \`${stagingTableUnqualified}\` AS (


I think what Nick means here is to split the sql creation into statements not the sql itself.

Tuseeq1 · 2026-06-12T09:18:09Z

+    INSERT (`id`,`field1`) VALUES (`id`,`field1`);
+END;
+
+DROP TABLE IF EXISTS `staging_table_temp_test_uuid`


If I read the execution_sql.ts file correctly, there should be a ";" at the end no? same in insert_overwrite_extend.sql.

Good catch. Fixed.

Reverted this change. concatenateQueries() helper function (in cli/api/dbadapters/tasks.ts) explicitly trims trailing whitespace and strips the final semicolon from every individual statement before joining them together with. Because of this stripping logic, the final sql that Dataform generates don't have a trailing semicolon at the very end.

Tuseeq1 · 2026-06-12T09:20:55Z

+      ...baseTable,
+      incrementalStrategy: dataform.IncrementalStrategy.INSERT_OVERWRITE,
+      bigquery: {
+        partitionBy: "DATE(ts)",


Do we have an example column that is not a function like Date. My concern is that we a re biased toward handling this method. so it its just a normal column name we might not handle it correctly.

I added additional test for normal column.

- updated proto with new parameters - added new tests - added validation for chosen incremental_strategies - added new insert_overwrite strategy logic

SuchodolskiEdvin assigned kolina and SuchodolskiEdvin Jun 2, 2026

SuchodolskiEdvin requested a review from a team as a code owner June 2, 2026 14:39

SuchodolskiEdvin requested review from zaptot and removed request for a team June 2, 2026 14:39

SuchodolskiEdvin force-pushed the insert_overwrite_incremental_strategy branch from 9f72a8c to 9724071 Compare June 2, 2026 15:13

SuchodolskiEdvin requested review from kolina and removed request for zaptot June 3, 2026 09:56

SuchodolskiEdvin force-pushed the insert_overwrite_incremental_strategy branch from 9724071 to 41d152d Compare June 3, 2026 13:46

kolina requested changes Jun 7, 2026

View reviewed changes

SuchodolskiEdvin force-pushed the insert_overwrite_incremental_strategy branch 4 times, most recently from f11f8f3 to 1942a72 Compare June 8, 2026 16:29

SuchodolskiEdvin requested review from Tuseeq1 and kolina June 10, 2026 09:04

Tuseeq1 requested changes Jun 12, 2026

View reviewed changes

SuchodolskiEdvin force-pushed the insert_overwrite_incremental_strategy branch 2 times, most recently from 7352b3d to a90b434 Compare June 22, 2026 15:37

Support incremental_strategy parameter and new insert_overwrite strategy

8f1bcb7

- updated proto with new parameters - added new tests - added validation for chosen incremental_strategies - added new insert_overwrite strategy logic

SuchodolskiEdvin force-pushed the insert_overwrite_incremental_strategy branch from a90b434 to 8f1bcb7 Compare June 22, 2026 16:03

SuchodolskiEdvin requested a review from Tuseeq1 June 22, 2026 16:30


		const result = runMainInVm(coreExecutionRequestFromPath(projectDir));

		expect(result.compile.compiledGraph.graphErrors.compilationErrors.length).greaterThan(0);

Conversation

SuchodolskiEdvin commented Jun 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SuchodolskiEdvin Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SuchodolskiEdvin Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SuchodolskiEdvin Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

SuchodolskiEdvin Jun 22, 2026 •

edited

Loading

SuchodolskiEdvin Jun 22, 2026 •

edited

Loading

SuchodolskiEdvin Jun 22, 2026 •

edited

Loading