Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

the partition expiry is set to 7 days by default for temp table #1066

Closed
2 tasks done
minhajpasha opened this issue Dec 22, 2023 · 0 comments
Closed
2 tasks done

the partition expiry is set to 7 days by default for temp table #1066

minhajpasha opened this issue Dec 22, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@minhajpasha
Copy link

Is this a new bug in dbt-bigquery?

  • I believe this is a new bug in dbt-bigquery
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I have primary model with below configuration

resolution_info.sql(model)
{{
config(
materialized='incremental',
incremental_strategy='insert_overwrite',
unique_key='resolution_id',
partition_by={"field":"event_date", "data_type":"DATE", "granularity":"day"},
partition_expiration_days=1825,
cluster_by=["local_event_date"],
on_schema_change='append_new_columns'
)
}}

This table is partition by event_date

We have got the requirement to refill the data from 2021(event_date) due to change calculation change which impacted the values in few columns.

We have loaded all the data in staging layer , as it's insert_overwrite it should normally replace the data from 2021 to till date.

The tmp table is getting created with partition expiry for 7 days and it stay's as empty as it's partition by event_date

temp table from logs

create or replace table `gcp-res-dev-`.`resolution`.`resolution_info__dbt_tmp`
partition by event_date
cluster by local_event_date
OPTIONS(
  description=""" Resolution""",

  labels=[('layer', 'primary'), ('created_by', 'dbt')],

  expiration_timestamp=TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 12 hour)
)
as (

with base as (
select
.....
when i had a look of table details in bq UI . i can see the table is created with 7 day's as partition expiry and the data is not getting replaced

Expected Behavior

as it's insert_overwrite it should normally replace the data from 2021 to till date.

Steps To Reproduce

the target should be insert_overwrite strategy table

Relevant log output

NA

Environment

Running with dbt=1.7.1
12:15:37  dbt version: 1.7.1
12:15:37  python version: 3.9.6
12:15:37  python path: /Library/Developer/CommandLineTools/usr/bin/python3
12:15:37  os info: macOS-13.6.2-arm64-arm-64bit

Additional Context

No response

@minhajpasha minhajpasha added bug Something isn't working triage labels Dec 22, 2023
@github-actions github-actions bot changed the title the partition expiry is set to 7 days by default [ADAP-1079] the partition expiry is set to 7 days by default Dec 22, 2023
@minhajpasha minhajpasha changed the title [ADAP-1079] the partition expiry is set to 7 days by default the partition expiry is set to 7 days by default for temp table Dec 22, 2023
@dbeatty10 dbeatty10 removed the triage label Jan 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants