dataform compile always emits the entire compiled graph. There is no way to ask
the CLI for the compiled SQL of a single model (or a subset of models) the way
--actions / --tags already work for dataform run.
I'd like dataform compile (and dataform compile --json) to accept the existing
selection flags so I can get the compiled SQL for just the action(s) I care about, e.g.:
dataform compile --actions my_model --json
dataform compile --tags daily --include_deps --json
This mirrors dbt's dbt compile --select my_model, which is a feature I rely on for
quickly inspecting the rendered SQL of one model without scrolling through the whole graph
or post-processing the JSON with jq.
Looking at code in this repo i understand that the compiler must evaluate the whole project because ref() resolution and dependency graph require every action to be registered. So this request is not about partial compilation its purely about filtering the compile ouput to the selected actions (similar to what prune does for run) its an UX improvement in CLI.
Possible Implementation
The filtering logic seems to already exists and is reused prune() in
cli/api/commands/prune.ts takes a CompiledGraph + a
RunConfig (actions, tags, includeDependencies, includeDependents) and returns a
filtered CompiledGraph. dataform run/build already call it.
The change is to make the compile command optionally apply prune() before printing:
Open question i have
I would like to work on this but have some questions to discuss
-
When a selector is present, should compilation
errors from unselected actions still fail the command / be printed? I lean toward keeping
graph-level errors as-is (the whole project still had to compile) but suppressing per-action
errors unrelated to the selection
-
f the selector matches nothing, should compile warn and
exit non-zero (like a "no actions" case) or just emit an empty graph?
dataform compilealways emits the entire compiled graph. There is no way to askthe CLI for the compiled SQL of a single model (or a subset of models) the way
--actions/--tagsalready work fordataform run.I'd like
dataform compile(anddataform compile --json) to accept the existingselection flags so I can get the compiled SQL for just the action(s) I care about, e.g.:
This mirrors dbt's
dbt compile --select my_model, which is a feature I rely on forquickly inspecting the rendered SQL of one model without scrolling through the whole graph
or post-processing the JSON with
jq.Looking at code in this repo i understand that the compiler must evaluate the whole project because
ref()resolution and dependency graph require every action to be registered. So this request is not about partial compilation its purely about filtering the compile ouput to the selected actions (similar to what prune does for run) its an UX improvement in CLI.Possible Implementation
The filtering logic seems to already exists and is reused
prune()incli/api/commands/prune.tstakes aCompiledGraph+ aRunConfig(actions,tags,includeDependencies,includeDependents) and returns afiltered
CompiledGraph.dataform run/buildalready call it.The change is to make the
compilecommand optionally applyprune()before printing:Open question i have
I would like to work on this but have some questions to discuss
When a selector is present, should compilation
errors from unselected actions still fail the command / be printed? I lean toward keeping
graph-level errors as-is (the whole project still had to compile) but suppressing per-action
errors unrelated to the selection
f the selector matches nothing, should
compilewarn andexit non-zero (like a "no actions" case) or just emit an empty graph?