Skip to content

Commit

Permalink
Updates
Browse files Browse the repository at this point in the history
  • Loading branch information
ilias1111 committed Nov 19, 2024
1 parent 731e2a4 commit b0a9245
Show file tree
Hide file tree
Showing 4 changed files with 38 additions and 40 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,12 @@ import {versions} from '@site/src/componentVersions';
<TabItem value="normalize" label="Snowplow Normalize">

<ReactMarkdown children={`
| snowplow-normalize version | dbt versions | BigQuery | Databricks | Redshift | Snowflake | Postgres |
| -------------------------------- | ----------------- | :------: | :--------: | :------: | :-------: | :------: |
| ${versions.dbtSnowplowNormalize} | >=1.4.0 to <2.0.0 ||||||
| 0.2.3 | >=1.3.0 to <2.0.0 ||||||
| 0.1.0 | >=1.0.0 to <2.0.0 ||||||
| snowplow-normalize version | dbt versions | BigQuery | Databricks | Redshift | Snowflake | Postgres | Spark |
| -------------------------------- | ----------------- | :------: | :--------: | :------: | :-------: | :------: | :------: |
| ${versions.dbtSnowplowNormalize} | >=1.4.0 to <2.0.0 |||||||
| 0.3.5 | >=1.4.0 to <2.0.0 |||||||
| 0.2.3 | >=1.3.0 to <2.0.0 |||||||
| 0.1.0 | >=1.0.0 to <2.0.0 |||||||
`} remarkPlugins={[remarkGfm]} />

</TabItem>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,44 +24,12 @@ Normalize in this context means [database normalization](https://en.wikipedia.or

**The package source code can be found in the [snowplow/dbt-snowplow-normalize repo](https://github.com/snowplow/dbt-snowplow-normalize), and the docs for the [macro design are here](https://snowplow.github.io/dbt-snowplow-normalize/#/overview/snowplow_normalize).**

## Package Configuration

### Partition Timestamp Configuration

The package uses a configurable partition timestamp column, controlled by the `snowplow__partition_tstamp` variable:

```yaml
vars:
snowplow__partition_tstamp: "collector_tstamp" # Default value
```
:::warning Important Note on Custom Partition Timestamps
If you change `snowplow__partition_tstamp` to a different column (e.g., "loader_tstamp"), you MUST ensure that this column is included in the `event_columns` list in your normalize configuration for each event. Failing to do so will cause the models to fail, as the partition column must be present in the normalized output.

Example configuration when using a custom partition timestamp:
```json
{
"events": [
{
"event_names": ["page_view"],
"event_columns": [
"domain_userid",
"loader_tstamp", // Must include your custom partition timestamp here
"app_id"
],
// ... rest of configuration
}
]
}
```
:::

The package provides [macros](https://docs.getdbt.com/docs/build/jinja-macros) and a python script that is used to generate your normalized events, filtered events, and users table for use within downstream ETL tools such as Census. See the [Model Design](#model-design) section for further details on these tables.

The package only includes the base incremental scratch model and does not have any derived models, instead it generates models in your project as if they were custom models you had built on top of the [Snowplow incremental tables](/docs/modeling-your-data/modeling-your-data-with-dbt/package-mechanics/incremental-processing/index.md), using the `_this_run` table as the base for new events to process each run. See the [configuration](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-configuration/index.md) section for the variables that apply to the incremental model.

:::note
The incremental model is simplified compared to the standard unified model, this package does not use sessions to identify which historic events to reprocess and just uses the `snowplow__partition_tstamp` (defaults to `collector_tstamp`) and package variables to identify which events to (re)process.
The incremental model is simplified compared to the standard unified model, this package does not use sessions to identify which historic events to reprocess and just uses the `snowplow__session_timestamp` (defaults to `collector_tstamp`) and package variables to identify which events to (re)process.
:::

## Model Design
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,37 @@ Depending on the use case it should either be the catalog (for Unity Catalog use

:::

### 10. Partition Timestamp Configuration

### 10. Run your model(s)
The package uses a configurable partition timestamp column, controlled by the `snowplow__partition_tstamp` variable:

```yaml
vars:
snowplow__partition_tstamp: "collector_tstamp" # Default value
```

:::warning Important Note on Custom Partition Timestamps
If you change `snowplow__partition_tstamp` to a different column (e.g., "loader_tstamp"), you MUST ensure that this column is included in the `event_columns` list in your normalize configuration for each event. Failing to do so will cause the models to fail, as the partition column must be present in the normalized output.

Example configuration when using a custom partition timestamp:
```json
{
"events": [
{
"event_names": ["page_view"],
"event_columns": [
"domain_userid",
"loader_tstamp", // Must include your custom partition timestamp here
"app_id"
],
// ... rest of configuration
}
]
}
```
:::

### 11. Run your model(s)

You can now run your models for the first time by running the below command (see the [operation](/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/index.md) page for more information on operation of the package):

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@
"description": "Determines which timestamp is used to process sessions of data"
},
"snowplow__partition_tstamp": {
"recommendFullRefresh": false,
"recommendFullRefresh": true,
"order": 56,
"consoleGroup": "advanced",
"type": "string",
Expand Down

0 comments on commit b0a9245

Please sign in to comment.