Bumps the CI test matrix to the latest supported Airflow and dbt releases, fixes the real compatibility breaks that surfaced along the way, and starts reducing the amount of dbt CLI surface we hardcode by hand. - #182
Open
millin wants to merge 14 commits into
Conversation
FreshnessTask.__init__ started requiring a catalogs argument in dbt-core 1.12, unlike its sibling ConfiguredTask subclasses which default it to None.
Airflow <3.3 (fixed upstream by apache/airflow#68840) schedules the FastAPI execution API app's lifespan startup via asyncio.run_coroutine_threadsafe without waiting for it, so a dag.test() task run can race ahead of app.state.svcs_registry being set and fail with AttributeError. We patch in_process_api_server() to hand out a pre-warmed transport that waits for lifespan startup, mirroring the upstream fix, since InProcessExecutionAPI is an attrs-slotted class we can't otherwise safely monkeypatch.
…ally BaseConfig hardcoded every dbt project-level behavior change flag as a dataclass field, which broke every time dbt added a new one (most recently require_source_and_semantic_model_names_without_spaces in 1.12) since dbt's own Flags object only sets the UPPERCASE attribute for flags absent from our config's vars(), while some dbt code paths read the lowercase one and raise AttributeError. Instead, read the current flag set from dbt.contracts.project.ProjectFlags.project_only_flags at runtime, so new dbt releases work without a change here. Also add an extra_flags escape hatch (mirroring the existing vars field) so any dbt flag or config option we don't otherwise model can still be passed through from the operator.
dbt-core <1.12 pins mashumaro<3.15, which doesn't support Python 3.14 (added only in mashumaro 3.17). Those combinations can't install and would just fail on every run. Also drops a stale beta-era comment now that dbt 1.12 has a stable release.
dbt's own CLI merges --flag/--no-flag into a single boolean option; the separate no_<flag> counterpart we've historically accepted was never something dbt itself needed, only kept for backwards compatibility. Warn callers so we can eventually drop it and shrink this parameter surface.
DagBag dropped the include_examples parameter in Airflow 3.3, breaking every matrix combination using that version. Add AIRFLOW_V_3_3_PLUS and only pass it on older Airflow. Also fixes two mypy errors that surfaced once the previously-broken mypy job condition started actually running mypy again: a possibly None connection type passed where dict.get expects str, and a conditional-import fallback assigning an incompatible type. While at it, point the mypy job condition at Python 3.14, the latest version in the matrix.
millin
force-pushed
the
chore/bump-all-the-versions
branch
from
September 8, 2026 16:45
fe225a5 to
f2546c1
Compare
Operators previously forwarded their entire __dict__ (vars(self)) to the dbt hook, which happened to work but meant any dbt CLI option not already modeled as a named parameter was silently dropped, and mixed in unrelated Airflow-internal attributes (task_id, dag, retries, ...) that just got filtered out downstream by field-name matching. Add a config_kwargs dict parameter to DbtBaseOperator: every operator now folds its own __init__ parameters into this single dict via a new _update_config_kwargs() helper, and execute() sends this (not vars(self)) to the hook. This is now the sole vehicle for reaching dbt, and named parameters without special handling (Jinja templating, or read elsewhere in the class) no longer get a redundant self.attr of their own. ConfigFactory.create_config() now routes any kwarg that doesn't match one of the target Config's fields into its extra_flags field instead of dropping it, so a dbt parameter we haven't modeled explicitly (e.g. a newer dbt release's addition) still reaches dbt. Updates 14 tests that called get_dbt_task_config(**vars(op)) directly to use **op.config_kwargs instead, matching what execute() now does.
millin
force-pushed
the
chore/bump-all-the-versions
branch
from
September 8, 2026 18:23
f2546c1 to
d0b9651
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Matrix updates
3.3.1(latest) and3.2.2(previous minor, kept for backward-compat coverage). Cloud Composer and MWAA both now track the latest release too, so no separate entries are needed for them.1.12.3,1.11.14,1.10.23(latest patch of each supported minor).python-version: '3.14'×dbt-version: '1.11.14'/'1.10.23': dbt-core<1.12pinsmashumaro<3.15, which has no Python 3.14 support (added only inmashumaro 3.17). This combination can't install, regardless of anything on our end — see dbt-labs/dbt-core#12098.mypyjob condition, which referenced Airflow/dbt versions that hadn't existed in the matrix for a while (somypywas silently never running).Compatibility fixes (dbt-core 1.12)
FreshnessTaskrequirescatalogs: dbt-core 1.12 added a requiredcatalogsargument toFreshnessTask.__init__, unlike siblingConfiguredTasksubclasses where it defaults toNone. AddedDBT_INSTALLED_GTE_1_12and passcatalogs=[]for this task specifically.BaseConfighardcoded every dbt behavior change flag (e.g.require_all_warnings_handled_by_warn_error) as a dataclass field. This breaks every time dbt adds a new one — most recentlyrequire_source_and_semantic_model_names_without_spacesin 1.12 — because dbt'sFlagsobject only sets the uppercase attribute for flags absent from our config'svars(), while some dbt code reads the lowercase one and raisesAttributeError. Replaced the hardcoded list with a runtime read ofdbt.contracts.project.ProjectFlags.project_only_flags, confirmed present with the same shape across all three matrix dbt versions. New dbt releases now pick up new flags automatically. Also added anextra_flagsescape hatch (mirroring the existingvarsfield) onBaseConfigand the operator, for anything we still don't model.Test infrastructure fix (Airflow <3.3)
InProcessExecutionAPIlifespan race: before apache/airflow#68840 (fixed in Airflow 3.3+),InProcessExecutionAPI.transportschedules the FastAPI execution-API app's lifespan startup viaasyncio.run_coroutine_threadsafewithout waiting for it, so adag.test()task run can race ahead ofapp.state.svcs_registrybeing set and fail withAttributeError. Added a workaround intests/conftest.pythat pre-warms the transport and waits for lifespan startup, mirroring the upstream fix, sinceInProcessExecutionAPIis anattrs-slotted class that can't otherwise be safely monkeypatched.Deprecations
no_<flag>counterpart of every boolean CLI option (no_introspect,no_defer,no_print, …) has never been something dbt itself needs — dbt's CLI merges--flag/--no-flaginto a single boolean option, so it was always purely our own invention. Left in place for backward compatibility, but now emits aDeprecationWarningpointing at<flag>=<value>instead, so it can eventually be removed and shrink this part of the public API.