Showcasing Feature Engineering with Data Engineering tools

Roadmap

• Feature Engineering with DBT
• Feature Engineering with Dataform Cloud
• Compatibility with Dataform Core
• DBT with python models

Download data in your BQ Environmnet

python tools/copy_bigquery_data.py --project vertex-ai-test-365213 --location  europe-west2   

Performing feature engineering with DBT

Resources:

• Learn more about dbt in the docs
• Learn how to setup DBT in your environment in this doc

Setup the environment

virtualenv venv && source venv/bin/activate && pip install -r requirements.txt

Create features

To build features:

(cd features/dbt_project && dbt run)

To run data assertions:

(cd features/dbt_project && dbt test)

To test the feature creation and ingestion process on a dummy feature store:

python tools/manual_ingestion_to_fs.py --config_path features/dbt_project/fs_config.yml

Performing feature engineering with Dataform core

The same code can be also executed in Dataform Cloud by copying the files in a workspace.

Resources:

• Learn more about dbt in the docs
• Learn how to setup DBT in your environment in this doc

Create features

npm i -g @dataform/cli@^2.0.0 
dataform install
dataform init-creds bigquery

Using the project

To build features:

(cd features/dataform_project && dataform run --full-refresh)

To run data assertions (no tests are defined at the moment):

(cd features/dataform_project && dataform test)

To test the feature creation and ingestion process on a dummy feature store: Install dependencies:

virtualenv venv && source venv/bin/activate && pip install -r requirements.txt

Perform ingestion:

python tools/manual_ingestion_to_fs.py --config_path features/dataform_project/fs_config.yml