Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(#27): Add dbt unit test brac #28

Merged
merged 44 commits into from
Nov 1, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
4cd3be2
Add tests to contact views working models
Sep 19, 2023
990dcd0
Add tests to use views working models
Sep 20, 2023
dbb7be1
feat: make model chnages and temporarily remove dashboard models
njogz Oct 5, 2023
209e467
feat: fix failing models
njogz Oct 9, 2023
5860411
Merge conflicts
Oct 18, 2023
6d19aa2
feat: fix failing models
njogz Oct 9, 2023
1ec49ce
Add users.yml to gitignore
Oct 18, 2023
be804cd
Merge with 30-fix-failing-brac-models
Oct 18, 2023
ff34b9b
Add tests for contact_views models
Oct 18, 2023
abf1b27
Add tests to useviews models
Oct 19, 2023
3e2d808
Fix some test
Oct 19, 2023
1a07e87
Add tests for main models
Oct 19, 2023
bec08a7
Add relationships and accepted_values tests
Oct 19, 2023
c23c2fd
Add relationships tests on contact views models
Oct 19, 2023
69a1942
Add relationships tests to main models
Oct 19, 2023
1c0f18c
Fix Did not find matching node for patch error
Oct 20, 2023
532fb73
Fix accepted values errors
Oct 20, 2023
3c3498a
Fix tests
Oct 20, 2023
261768d
Fixing failing tests
Oct 20, 2023
63feb64
Delete some failing tests
Oct 20, 2023
5ce7ac8
First views model test
Oct 20, 2023
349016c
Add useviews model tests
Oct 20, 2023
0e61edf
Change the way columns are referenced in couchdb
Oct 20, 2023
5da2bbb
Trying to see if the problem are the seeds
Oct 20, 2023
77d71fd
Add test to pncview_actual_enrrollments model
Oct 20, 2023
6ae98d8
Add test for form_metadata model
Oct 20, 2023
efc1e0a
Add test for impactview_month
Oct 20, 2023
ac99bb4
Add test to formview_inmunization_follow_up model
Oct 20, 2023
c77e687
Add test for formview_assessment_follow_up model
Oct 20, 2023
7acf844
Add test for formview_muac_follow_up
Oct 20, 2023
eba6e13
Add test on couples of views models
Oct 20, 2023
b37e862
Fix views.yml duplicate
Oct 20, 2023
a7eaedd
Add more test on views models
Oct 20, 2023
d68de81
Add more test to views models
Oct 20, 2023
6af0f70
Delete failed tests
Oct 20, 2023
77d9d1b
Add first test for impact anc models
Oct 20, 2023
23d8ad6
Delete exclude command from run dbt
Oct 20, 2023
55616a7
Revert deletion of failing tests
Oct 20, 2023
9531756
Reverse change
Oct 20, 2023
82b612a
reverse deletions
Oct 20, 2023
d7a3929
Fix brac name
Oct 20, 2023
ef62474
Finish test for views models
Oct 20, 2023
184fa45
Move and rename impact test file. Implement impact tests
Oct 23, 2023
ffc4fdd
Delete commented lines
Nov 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,7 @@ dbt_packages/
logs/
*/.env
*/dbt_packages/
tests/.user.yml
dbt-env/*
.user.yml
dbt-env/*
35 changes: 18 additions & 17 deletions dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,34 +1,35 @@
# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: "pipeline"
version: "0.0.1"
name: 'pipeline'
version: '0.0.1'
config-version: 2

vars:
# The `start_date` variable will be accessible in all resources
end_date: '2023-06-01'

# This setting configures which "profile" dbt uses for this project.
profile: "default"
profile: 'default'

# These configurations specify where dbt should look for different types of files.
# The `source-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
model-paths: ['models']
analysis-paths: ['analysis']
test-paths: ['tests']
seed-paths: ['seeds']
macro-paths: ['macros']
snapshot-paths: ['snapshots']

target-path: "target" # directory which will store compiled SQL files
target-path: 'target' # directory which will store compiled SQL files
clean-targets: # directories to be removed by `dbt clean`
- "target"
- "dbt_modules"
- 'target'
- 'dbt_modules'

# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models
vars:
end_date: "2020-12-12"

# In this example config, we tell dbt to build all models in the example/ directory
# as tables. These settings can be overridden in the individual model files
# using the `{{ config(...) }}` macro.
models:
type_partitions:
root:
Expand Down
74 changes: 74 additions & 0 deletions models/brac/brac.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
version: 2

models:
- name: brac_hmis
columns:
- name: uuid
tests:
- not_null
- unique
- relationships:
to: ref('contactview_chp')
field: uuid
- name: households_registered
tests:
- not_null
- name: num_of_hh_visited
tests:
- not_null
- name: num_of_hh_bed_nets
tests:
- not_null
- name: num_of_hh_latrines
tests:
- not_null
- name: num_of_hh_safe_water
tests:
- not_null
- name: num_of_comm_members
tests:
- not_null
- name: num_of_health_forum
tests:
- not_null
- name: num_pregnant_care
tests:
- not_null
- name: num_pregnant_identified
tests:
- not_null
- name: num_facility_deliveries
tests:
- not_null
- name: num_home_deliveries
tests:
- not_null
- name: brac-kpis
columns:
- name: chp_uuid
tests:
- not_null
- relationships:
to: ref('contactview_chp')
field: uuid
- name: count_all_assessments
tests:
- not_null
- name: u5_malaria_treatment
tests:
- not_null
- name: u5_diarrhea_treatment
tests:
- not_null
- name: u5_pneumonia_treatment
tests:
- not_null
- name: u1_malaria_treatment
tests:
- not_null
- name: u1_diarrhea_treatment
tests:
- not_null
- name: u1_pneumonia_treatment
tests:
- not_null
143 changes: 143 additions & 0 deletions models/brac/contact_views/contact_views.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
version: 2

models:
- name: contactview_person
columns:
- name: uuid
tests:
- not_null
- unique
- relationships:
to: ref('couchdb')
field: doc ->> '_id'
- name: name
tests:
- not_null
- name: patient_id
tests:
- not_null
- name: contactview_person_fields
columns:
- name: uuid
tests:
- not_null
- unique
- relationships:
to: ref('contactview_metadata')
field: uuid
tests:
- dbt_utils.fewer_rows_than:
compare_model: ref('contactview_metadata')
- name: contactview_chw
columns:
- name: uuid
tests:
- not_null
- unique
- relationships:
to: ref('contactview_person_fields')
field: uuid
tests:
- dbt_utils.fewer_rows_than:
compare_model: ref('contactview_metadata')
- dbt_utils.fewer_rows_than:
compare_model: ref('contactview_person_fields')
- name: contactview_chp
columns:
- name: uuid
tests:
- not_null
- unique
- relationships:
to: ref('contactview_chw')
field: uuid
- name: branch_uuid
tests:
- not_null
- name: supervisor_uuid
tests:
- not_null
tests:
- dbt_utils.fewer_rows_than:
compare_model: ref('contactview_metadata')
- dbt_utils.fewer_rows_than:
compare_model: ref('contactview_person_fields')
- name: contactview_muted
columns:
- name: contact_uuid
tests:
- not_null
- unique
- relationships:
to: ref('contactview_metadata')
field: uuid
- name: type
tests:
- not_null
- accepted_values:
values:
[contact, clinic, district_hospital, health_center, person]
- name: mute_status
tests:
- not_null
- name: date
tests:
- not_null
tests:
- dbt_utils.fewer_rows_than:
compare_model: ref('contactview_metadata')
- name: contactview_hospital
columns:
- name: uuid
tests:
- not_null
- unique
- relationships:
to: ref('contactview_metadata')
field: uuid
tests:
- dbt_utils.fewer_rows_than:
compare_model: ref('contactview_metadata')
- name: contactview_hierarchy
columns:
- name: branch_uuid
tests:
- not_null
- relationships:
to: ref('contactview_branch')
field: uuid
- name: chw_uuid
tests:
- not_null
- relationships:
to: ref('contactview_chp')
field: uuid
- name: chw_area_uuid
tests:
- not_null
- name: contactview_branch
columns:
- name: uuid
tests:
- not_null
- unique
- relationships:
to: ref('contactview_hospital')
field: uuid
- name: contactview_clinic
columns:
- name: uuid
tests:
- not_null
- relationships:
to: ref('contactview_metadata')
field: uuid
- name: chw_uuid
tests:
- not_null
- relationships:
to: ref('contactview_chw')
field: uuid
tests:
- dbt_utils.fewer_rows_than:
compare_model: ref('contactview_metadata')
3 changes: 1 addition & 2 deletions models/brac/contact_views/contactview_chp.sql
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,7 @@ SELECT
chp.doc #>> '{chp_profile, g_language, mother_tongue}'::text[] AS mother_tongue,
chp.doc #>> '{chp_profile, g_language, other_languages}'::text[] AS other_languages,
chp.doc #>> '{chp_profile, g_other_details, incentives}'::text[] AS incentives,
chp.doc #>> '{chp_profile, g_other_details, chp_services}'::text[] AS chp_services,
contactview_chw."@timestamp" AS "@timestamp"
chp.doc #>> '{chp_profile, g_other_details, chp_services}'::text[] AS chp_services
FROM
{{ ref("contactview_chw") }}
JOIN {{ ref("raw_contacts") }} ON contactview_chw.area_uuid = (raw_contacts.doc ->> '_id'::text)
Expand Down
3 changes: 1 addition & 2 deletions models/brac/contact_views/contactview_chw.sql
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@ SELECT chw.name,
pplfields.date_of_birth,
pplfields.parent_type,
chwarea.uuid AS area_uuid,
chwarea.parent_uuid AS branch_uuid,
pplfields."@timestamp" AS "@timestamp"
chwarea.parent_uuid AS branch_uuid
FROM {{ ref("contactview_person_fields") }} pplfields
JOIN {{ ref("contactview_metadata") }} chw ON chw.uuid = pplfields.uuid
JOIN {{ ref("contactview_metadata") }} chwarea ON chw.parent_uuid = chwarea.uuid
Expand Down
3 changes: 1 addition & 2 deletions models/brac/contact_views/contactview_person_fields.sql
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,7 @@ SELECT person.uuid,
person.phone,
person.phone2,
person.date_of_birth,
parent.type AS parent_type,
person."@timestamp" AS "@timestamp"
parent.type AS parent_type
FROM {{ ref("contactview_metadata") }} person
LEFT JOIN {{ ref("contactview_metadata") }} parent ON person.parent_uuid = parent.uuid
WHERE person.type = 'person'::text
2 changes: 1 addition & 1 deletion models/brac/impact/ANC/impact_ancview_danger_sign.sql
Original file line number Diff line number Diff line change
Expand Up @@ -43,4 +43,4 @@ SELECT
reported_by_parent,
reported
FROM
danger_sign_CTE;
danger_sign_CTE
2 changes: 1 addition & 1 deletion models/brac/impact/ANC/impact_ancview_delivery.sql
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,4 @@ LEFT JOIN preg ON (pnc.reported > preg.reported AND pnc.reported < (preg.reporte
WHERE
pnc.follow_up_count = '1'
AND (pnc.pregnancy_outcome IN (VALUES ('healthy'), ('still_birth')))
AND pnc.patient_id IS NOT NULL AND pnc.patient_id <> '';
AND pnc.patient_id IS NOT NULL AND pnc.patient_id <> ''
33 changes: 0 additions & 33 deletions models/brac/impact/ANC/impact_ancview_pregnancy.sql

This file was deleted.

2 changes: 1 addition & 1 deletion models/brac/impact/ANC/impact_ancview_pregnancy_visit.sql
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,4 @@ SELECT
FROM
{{ ref('useview_visit') }}
WHERE
visit_type = 'anc';
visit_type = 'anc'
Loading
Loading