You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have got the requirement to refill the data from 2021(event_date) due to change calculation change which impacted the values in few columns.
We have loaded all the data in staging layer , as it's insert_overwrite it should normally replace the data from 2021 to till date.
The tmp table is getting created with partition expiry for 7 days and it stay's as empty as it's partition by event_date
temp table from logs
create or replace table `gcp-res-dev-`.`resolution`.`resolution_info__dbt_tmp`
partition by event_date
cluster by local_event_date
OPTIONS(
description=""" Resolution""",
labels=[('layer', 'primary'), ('created_by', 'dbt')],
expiration_timestamp=TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 12 hour)
)
as (
with base as (
select
.....
when i had a look of table details in bq UI . i can see the table is created with 7 day's as partition expiry and the data is not getting replaced
Expected Behavior
as it's insert_overwrite it should normally replace the data from 2021 to till date.
Steps To Reproduce
the target should be insert_overwrite strategy table
Relevant log output
NA
Environment
Running with dbt=1.7.1
12:15:37 dbt version: 1.7.1
12:15:37 python version: 3.9.6
12:15:37 python path: /Library/Developer/CommandLineTools/usr/bin/python3
12:15:37 os info: macOS-13.6.2-arm64-arm-64bit
Additional Context
No response
The text was updated successfully, but these errors were encountered:
github-actionsbot
changed the title
the partition expiry is set to 7 days by default
[ADAP-1079] the partition expiry is set to 7 days by default
Dec 22, 2023
minhajpasha
changed the title
[ADAP-1079] the partition expiry is set to 7 days by default
the partition expiry is set to 7 days by default for temp table
Dec 22, 2023
Is this a new bug in dbt-bigquery?
Current Behavior
I have primary model with below configuration
resolution_info.sql(model)
{{
config(
materialized='incremental',
incremental_strategy='insert_overwrite',
unique_key='resolution_id',
partition_by={"field":"event_date", "data_type":"DATE", "granularity":"day"},
partition_expiration_days=1825,
cluster_by=["local_event_date"],
on_schema_change='append_new_columns'
)
}}
This table is partition by event_date
We have got the requirement to refill the data from 2021(event_date) due to change calculation change which impacted the values in few columns.
We have loaded all the data in staging layer , as it's insert_overwrite it should normally replace the data from 2021 to till date.
The tmp table is getting created with partition expiry for 7 days and it stay's as empty as it's partition by event_date
temp table from logs
with base as (
select
.....
when i had a look of table details in bq UI . i can see the table is created with 7 day's as partition expiry and the data is not getting replaced
Expected Behavior
as it's insert_overwrite it should normally replace the data from 2021 to till date.
Steps To Reproduce
the target should be insert_overwrite strategy table
Relevant log output
Environment
Additional Context
No response
The text was updated successfully, but these errors were encountered: