Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: dbt-labs/dbt-core
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.9.1
Choose a base ref
...
head repository: dbt-labs/dbt-core
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.9.2
Choose a head ref
  • 8 commits
  • 15 files changed
  • 7 contributors

Commits on Dec 16, 2024

  1. [Automated] Merged prep-release/1.9.1_12358580707 into target 1.9.lat…

    …est during release process
    FishtownBuildBot authored Dec 16, 2024
    Copy the full SHA
    75aeafe View commit details

Commits on Dec 18, 2024

  1. Fix yaml snapshot specification with data tests (#11156) (#11167)

    (cherry picked from commit 6076cf7)
    
    Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
    github-actions[bot] and gshank authored Dec 18, 2024
    Copy the full SHA
    c6e4471 View commit details

Commits on Dec 19, 2024

  1. Check modified contracts when doing state:modified (#11161) (#11168)

    (cherry picked from commit 88e953e)
    
    Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
    github-actions[bot] and gshank authored Dec 19, 2024
    Copy the full SHA
    396fbe5 View commit details
  2. Fix for dbt_project.yml "tests" config resulting in incorrect state:m…

    …odified (#11166) (#11169)
    
    (cherry picked from commit bf18b59)
    
    Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
    github-actions[bot] and gshank authored Dec 19, 2024
    Copy the full SHA
    6a9e35a View commit details

Commits on Jan 8, 2025

  1. Always emit warning when microbatch models lack any filtered input no…

    …de (#11196) (#11199)
    
    * Update `TestMicrobatchWithInputWithoutEventTime` to check running again raises warning
    
    The first time the project is run, the appropriate warning about inputs is raised. However,
    the warning is only being raised when a full parse happens. When partial parsing happens
    the warning isn't getting raised. In the next commit we'll fix this issue. This commit updates
    the test to show that the second run (with partial parsing) doesn't raise the update, and thus
    the test fails.
    
    * Update manifest loading to _always_ check microbatch model inputs
    
    Of note we are at the point where multiple validations are iterating
    all of the nodes in a manifest. We should refactor these _soon_ such that
    we are not iterating over the same list multiple times.
    
    * Add changie doc
    
    (cherry picked from commit 2eb1a5c)
    
    Co-authored-by: Quigley Malcolm <QMalcolm@users.noreply.github.com>
    github-actions[bot] and QMalcolm authored Jan 8, 2025
    Copy the full SHA
    42a58bb View commit details

Commits on Jan 20, 2025

  1. Fix microbatch dbt list --output JSON (#11187) (#11203)

    MichelleArk authored Jan 20, 2025
    Copy the full SHA
    47b5109 View commit details

Commits on Jan 28, 2025

  1. point to the dbt-adapters subdirectory post-monorepo migration (#11244)…

    … (#11245)
    
    (cherry picked from commit 48d9afa)
    
    Co-authored-by: Mike Alfare <13974384+mikealfare@users.noreply.github.com>
    github-actions[bot] and mikealfare authored Jan 28, 2025
    Copy the full SHA
    d87b350 View commit details

Commits on Jan 29, 2025

  1. Copy the full SHA
    cb95373 View commit details
2 changes: 1 addition & 1 deletion .bumpversion.cfg
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.9.1
current_version = 1.9.2
parse = (?P<major>[\d]+) # major version number
\.(?P<minor>[\d]+) # minor version number
\.(?P<patch>[\d]+) # patch version number
12 changes: 12 additions & 0 deletions .changes/1.9.2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
## dbt-core 1.9.2 - January 29, 2025

### Fixes

- Error writing generic test at run time ([#11110](https://github.com/dbt-labs/dbt-core/issues/11110))
- Run check_modified_contract for state:modified ([#11034](https://github.com/dbt-labs/dbt-core/issues/11034))
- Fix unrendered_config for tests from dbt_project.yml ([#11146](https://github.com/dbt-labs/dbt-core/issues/11146))
- Ensure warning about microbatch lacking filter inputs is always fired ([#11159](https://github.com/dbt-labs/dbt-core/issues/11159))
- Fix microbatch dbt list --output json ([#10556](https://github.com/dbt-labs/dbt-core/issues/10556), [#11098](https://github.com/dbt-labs/dbt-core/issues/11098))

### Contributors
- [@internetcoffeephone](https://github.com/internetcoffeephone) ([#10556](https://github.com/dbt-labs/dbt-core/issues/10556), [#11098](https://github.com/dbt-labs/dbt-core/issues/11098))
15 changes: 14 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -5,6 +5,20 @@
- "Breaking changes" listed under a version may require action from end users or external maintainers when upgrading to that version.
- Do not edit this file directly. This file is auto-generated using [changie](https://github.com/miniscruff/changie). For details on how to document a change, see [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-changelog-entry)

## dbt-core 1.9.2 - January 29, 2025

### Fixes

- Error writing generic test at run time ([#11110](https://github.com/dbt-labs/dbt-core/issues/11110))
- Run check_modified_contract for state:modified ([#11034](https://github.com/dbt-labs/dbt-core/issues/11034))
- Fix unrendered_config for tests from dbt_project.yml ([#11146](https://github.com/dbt-labs/dbt-core/issues/11146))
- Ensure warning about microbatch lacking filter inputs is always fired ([#11159](https://github.com/dbt-labs/dbt-core/issues/11159))
- Fix microbatch dbt list --output json ([#10556](https://github.com/dbt-labs/dbt-core/issues/10556), [#11098](https://github.com/dbt-labs/dbt-core/issues/11098))

### Contributors
- [@internetcoffeephone](https://github.com/internetcoffeephone) ([#10556](https://github.com/dbt-labs/dbt-core/issues/10556), [#11098](https://github.com/dbt-labs/dbt-core/issues/11098))


## dbt-core 1.9.1 - December 16, 2024

### Fixes
@@ -18,7 +32,6 @@
### Contributors
- [@dave-connors-3](https://github.com/dave-connors-3) ([#10230](https://github.com/dbt-labs/dbt-core/issues/10230))


## dbt-core 1.9.0 - December 09, 2024

### Breaking Changes
3 changes: 3 additions & 0 deletions core/dbt/config/project.py
Original file line number Diff line number Diff line change
@@ -199,6 +199,9 @@ def load_raw_project(project_root: str) -> Dict[str, Any]:
if not isinstance(project_dict, dict):
raise DbtProjectError(f"{DBT_PROJECT_FILE_NAME} does not parse to a dictionary")

if "tests" in project_dict and "data_tests" not in project_dict:
project_dict["data_tests"] = project_dict.pop("tests")

return project_dict


17 changes: 14 additions & 3 deletions core/dbt/graph/selector_methods.py
Original file line number Diff line number Diff line change
@@ -680,17 +680,24 @@ def check_macros_modified(self, node):
def check_modified_content(
self, old: Optional[SelectorTarget], new: SelectorTarget, adapter_type: str
) -> bool:
different_contents = False
if isinstance(
new,
(SourceDefinition, Exposure, Metric, SemanticModel, UnitTestDefinition, SavedQuery),
):
# these all overwrite `same_contents`
different_contents = not new.same_contents(old) # type: ignore
else:
elif new: # because we also pull in deleted/disabled nodes, this could be None
different_contents = not new.same_contents(old, adapter_type) # type: ignore

upstream_macro_change = self.check_macros_modified(new)
return different_contents or upstream_macro_change

check_modified_contract = False
if isinstance(old, ModelNode):
func = self.check_modified_contract("same_contract", adapter_type)
check_modified_contract = func(old, new)

return different_contents or upstream_macro_change or check_modified_contract

def check_unmodified_content(
self, old: Optional[SelectorTarget], new: SelectorTarget, adapter_type: str
@@ -792,7 +799,11 @@ def search(self, included_nodes: Set[UniqueId], selector: str) -> Iterator[Uniqu
yield unique_id

# checkers that can handle removed nodes
if checker.__name__ in ["check_modified_contract"]:
if checker.__name__ in [
"check_modified_contract",
"check_modified_content",
"check_unmodified_content",
]:
# ignore included_nodes, since those cannot contain removed nodes
for previous_unique_id, previous_node in manifest.nodes.items():
# detect removed (deleted, renamed, or disabled) nodes
38 changes: 23 additions & 15 deletions core/dbt/parser/manifest.py
Original file line number Diff line number Diff line change
@@ -513,6 +513,7 @@ def load(self) -> Manifest:
self.check_for_spaces_in_resource_names()
self.check_for_microbatch_deprecations()
self.check_forcing_batch_concurrency()
self.check_microbatch_model_has_a_filtered_input()

return self.manifest

@@ -1472,21 +1473,6 @@ def check_valid_microbatch_config(self):
f"Microbatch model '{node.name}' optional 'concurrent_batches' config must be of type `bool` if specified, but got: {type(concurrent_batches)})."
)

# Validate upstream node event_time (if configured)
has_input_with_event_time_config = False
for input_unique_id in node.depends_on.nodes:
input_node = self.manifest.expect(unique_id=input_unique_id)
input_event_time = input_node.config.event_time
if input_event_time:
if not isinstance(input_event_time, str):
raise dbt.exceptions.ParsingError(
f"Microbatch model '{node.name}' depends on an input node '{input_node.name}' with an 'event_time' config of invalid (non-string) type: {type(input_event_time)}."
)
has_input_with_event_time_config = True

if not has_input_with_event_time_config:
fire_event(MicrobatchModelNoEventTimeInputs(model_name=node.name))

def check_forcing_batch_concurrency(self) -> None:
if self.manifest.use_microbatch_batches(project_name=self.root_project.project_name):
adapter = get_adapter(self.root_project)
@@ -1508,6 +1494,28 @@ def check_forcing_batch_concurrency(self) -> None:
)
)

def check_microbatch_model_has_a_filtered_input(self):
if self.manifest.use_microbatch_batches(project_name=self.root_project.project_name):
for node in self.manifest.nodes.values():
if (
node.config.materialized == "incremental"
and node.config.incremental_strategy == "microbatch"
):
# Validate upstream node event_time (if configured)
has_input_with_event_time_config = False
for input_unique_id in node.depends_on.nodes:
input_node = self.manifest.expect(unique_id=input_unique_id)
input_event_time = input_node.config.event_time
if input_event_time:
if not isinstance(input_event_time, str):
raise dbt.exceptions.ParsingError(
f"Microbatch model '{node.name}' depends on an input node '{input_node.name}' with an 'event_time' config of invalid (non-string) type: {type(input_event_time)}."
)
has_input_with_event_time_config = True

if not has_input_with_event_time_config:
fire_event(MicrobatchModelNoEventTimeInputs(model_name=node.name))

def write_perf_info(self, target_path: str):
path = os.path.join(target_path, PERF_INFO_FILE_NAME)
write_file(path, json.dumps(self._perf_info, cls=dbt.utils.JSONEncoder, indent=4))
9 changes: 8 additions & 1 deletion core/dbt/parser/schemas.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import datetime
import pathlib
import time
from abc import ABCMeta, abstractmethod
from dataclasses import dataclass, field
@@ -289,9 +290,15 @@ def _add_yaml_snapshot_nodes_to_manifest(
parser = SnapshotParser(self.project, self.manifest, self.root_project)
fqn = parser.get_fqn_prefix(block.path.relative_path)
fqn.append(snapshot["name"])

compiled_path = str(
pathlib.PurePath("").joinpath(
block.path.relative_path, snapshot["name"] + ".sql"
)
)
snapshot_node = parser._create_parsetime_node(
block,
self.get_compiled_path(block),
compiled_path,
parser.initial_config(fqn),
fqn,
snapshot["name"],
4 changes: 3 additions & 1 deletion core/dbt/task/list.py
Original file line number Diff line number Diff line change
@@ -17,6 +17,7 @@
from dbt.node_types import NodeType
from dbt.task.base import resource_types_from_args
from dbt.task.runnable import GraphRunnableTask
from dbt.utils import JSONEncoder
from dbt_common.events.contextvars import task_contextvars
from dbt_common.events.functions import fire_event, warn_or_error
from dbt_common.events.types import PrintEvent
@@ -142,7 +143,8 @@ def generate_json(self):
if self.args.output_keys
else k in self.ALLOWED_KEYS
)
}
},
cls=JSONEncoder,
)

def generate_paths(self) -> Iterator[str]:
2 changes: 1 addition & 1 deletion core/dbt/version.py
Original file line number Diff line number Diff line change
@@ -226,5 +226,5 @@ def _get_adapter_plugin_names() -> Iterator[str]:
yield plugin_name


__version__ = "1.9.1"
__version__ = "1.9.2"
installed = get_installed_version()
2 changes: 1 addition & 1 deletion core/setup.py
Original file line number Diff line number Diff line change
@@ -25,7 +25,7 @@


package_name = "dbt-core"
package_version = "1.9.1"
package_version = "1.9.2"
description = """With dbt, data analysts and engineers can build analytics \
the way engineers build applications."""

2 changes: 1 addition & 1 deletion dev-requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
git+https://github.com/dbt-labs/dbt-adapters.git@main
git+https://github.com/dbt-labs/dbt-adapters.git@main#subdirectory=dbt-adapters
git+https://github.com/dbt-labs/dbt-adapters.git@main#subdirectory=dbt-tests-adapter
git+https://github.com/dbt-labs/dbt-common.git@main
git+https://github.com/dbt-labs/dbt-postgres.git@main
17 changes: 17 additions & 0 deletions tests/functional/defer_state/test_modified_state.py
Original file line number Diff line number Diff line change
@@ -676,6 +676,15 @@ def test_delete_unversioned_contracted_model(self, project):
assert expected_warning in logs
assert expected_change in logs

# the same but for general-purpose state:modified
_, logs = run_dbt_and_capture(
["run", "--models", "state:modified.contract", "--state", "./state"]
)
expected_warning = "While comparing to previous project state, dbt detected a breaking change to an unversioned model"
expected_change = "Contracted model 'model.test.table_model' was deleted or renamed"
assert expected_warning in logs
assert expected_change in logs


class TestDeleteVersionedContractedModel(BaseModifiedState):
MODEL_UNIQUE_ID = "model.test.table_model.v1"
@@ -697,6 +706,14 @@ def test_delete_versioned_contracted_model(self, project):
e.value
)

# the same but for general-purpose state:modified
with pytest.raises(ContractBreakingChangeError) as e:
run_dbt(["run", "--models", "state:modified", "--state", "./state"])

assert "Contracted model 'model.test.table_model.v1' was deleted or renamed." in str(
e.value
)


class TestDisableUnversionedContractedModel(BaseModifiedState):
MODEL_UNIQUE_ID = "model.test.table_model"
45 changes: 45 additions & 0 deletions tests/functional/defer_state/test_unrendered_config.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
import pytest

from dbt.tests.util import run_dbt

dbt_project_update = """
models:
my_dbt_project:
+materialized: table
tests:
+store_failures: true
"""

foo_sql = """
select 1 as id
"""

schema_yml = """
models:
- name: foo
columns:
- name: id
tests:
- unique
"""


class TestGenericTestUnrenderedConfig:
@pytest.fixture(scope="class")
def project_config_update(self):
return dbt_project_update

@pytest.fixture(scope="class")
def models(self):
return {
"foo.sql": foo_sql,
"schema.yml": schema_yml,
}

def test_unrendered_config(self, project):
manifest = run_dbt(["parse"])
assert manifest
test_node_id = "test.test.unique_foo_id.fa8c520a2e"
test_node = manifest.nodes[test_node_id]
assert test_node.unrendered_config == {"store_failures": True}
19 changes: 18 additions & 1 deletion tests/functional/microbatch/test_microbatch.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import json
from unittest import mock

import pytest
@@ -15,6 +16,7 @@
MicrobatchMacroOutsideOfBatchesDeprecation,
MicrobatchModelNoEventTimeInputs,
)
from dbt.tests.fixtures.project import TestProjInfo
from dbt.tests.util import (
get_artifact,
patch_microbatch_end_time,
@@ -333,6 +335,19 @@ class TestMicrobatchCLIBuild(TestMicrobatchCLI):
CLI_COMMAND_NAME = "build"


class TestMicrobatchCLIRunOutputJSON(BaseMicrobatchTest):
def test_list_output_json(self, project: TestProjInfo):
"""Test whether the command `dbt list --output json` works"""
model_catcher = EventCatcher(event_to_catch=LogModelResult)
batch_catcher = EventCatcher(event_to_catch=LogBatchResult)

_, microbatch_json = run_dbt(
["list", "--output", "json"], callbacks=[model_catcher.catch, batch_catcher.catch]
)
microbatch_dict = json.loads(microbatch_json)
assert microbatch_dict["config"]["begin"] == "2020-01-01T00:00:00"


class TestMicroBatchBoundsDefault(BaseMicrobatchTest):
def test_run_with_event_time(self, project):
# initial run -- backfills all data
@@ -464,9 +479,11 @@ def test_run_with_event_time(self, project):
assert len(catcher.caught_events) == 1

# our partition grain is "day" so running the same day without new data should produce the same results
catcher.caught_events = []
with patch_microbatch_end_time("2020-01-03 14:57:00"):
run_dbt(["run"])
run_dbt(["run"], callbacks=[catcher.catch])
self.assert_row_count(project, "microbatch_model", 3)
assert len(catcher.caught_events) == 1

# add next two days of data
test_schema_relation = project.adapter.Relation.create(
8 changes: 8 additions & 0 deletions tests/functional/snapshots/fixtures.py
Original file line number Diff line number Diff line change
@@ -301,6 +301,10 @@
updated_at: updated_at
meta:
owner: 'a_owner'
columns:
- name: id
data_tests:
- not_null
"""

snapshots_pg__snapshot_mod_yml = """
@@ -313,6 +317,10 @@
updated_at: updated_at
meta:
owner: 'b_owner'
columns:
- name: id
data_tests:
- not_null
"""

snapshots_pg__snapshot_no_target_schema_sql = """