Release/v3.7.1 sprint 111

.circleci/build-and-test/commands.yml

@@ -25,7 +25,7 @@
             fi
             echo "export CURRENT_FLAG=$CURRENT_FLAG" >> $BASH_ENV
       - run:
-          name: Upload code coverage report if target branch
+          name: Upload code coverage report of target branch
           command: codecov -t "$CODECOV_TOKEN" -f <<parameters.coverage-report>> -F "$CURRENT_FLAG"
 
   install-nodejs-machine:

.gitconfig

@@ -14,3 +14,4 @@
 	allowed = .git/config:.*
 	allowed = .gitconfig:.*
 	allowed = .*DJANGO_SECRET_KEY=.*
+	allowed = ./tdrs-backend/plg/loki/manifest.yml:*

.gitignore

@@ -109,4 +109,9 @@ cypress.env.json
 
 # Patches
 *.patch
+
+# Logs
+*.log
+
+# DB seeds
 tdrs-backend/*.pg

Taskfile.yml

@@ -2,11 +2,6 @@ version: '3'
 
 tasks:
 
-  upload-kibana-objs:
-    desc: Upload dashboards to Kibana server
-    cmds:
-      - 'curl -X POST localhost:5601/api/saved_objects/_import -H "kbn-xsrf: true" --form file=@tdrs-backend/tdpservice/search_indexes/kibana_saved_objs.ndjson'
-
   create-network:
     desc: Create the external network
     cmds:
@@ -251,7 +246,7 @@ tasks:
     desc: Open a shell in the frontend container
     dir: tdrs-frontend
     cmds:
-      - docker-compose -f docker-compose.yml exec tdp-frontend sh
+      - docker-compose -f docker-compose.yml exec tdp-frontend bash
 
   up:
     desc: Start both frontend and backend web servers
@@ -268,4 +263,4 @@ tasks:
   help:
     desc: Show this help message
     cmds:
-      - task --list
+      - task --list

codecov.yml

@@ -42,4 +42,7 @@ flags:
     carryforward: true
 
 ignore:
-  - "tdrs-backend/tdpservice/scheduling/db_backup.py"
+  - "tdrs-backend/tdpservice/scheduling/db_backup.py"
+  - "tdrs-backend/tdpservice/search_indexes/admin/mulitselect_filter.py"
+  - "tdrs-backend/tdpservice/email/helpers/account_access_requests.py"
+  - "tdrs-backend/tdpservice/search_indexes/admin/filters.py"

docs/Technical-Documentation/Architecture-Decision-Record/022-app-monitoring.md

@@ -0,0 +1,68 @@
+# 22. Monitoring Application Health and Performance
+
+Date: 2024-09-30
+
+## Status
+
+Pending
+
+## Context
+Historic feedback highlighted an ongoing desire for improved alerting and monitoring mechanisms, particularly originating in issue [#831](https://github.com/raft-tech/TANF-app/issues/831) circa 2021. Currently, our cloud platform has limited logging features and user interface issues leading to a "blindness" to errors and stack traces that have occurred, ultimately impairing our ability to maintain system stability; additionally, the existing dashboards only offer live performance data lacking data over time or any archives. Without context for either performance or system logging, determination of anomalous or erroneous system behavior is not possible.
+
+Additionally, we have experienced critical blocking issues related to our updates to both Elasticsearch (ES) and PostgreSQL, which have compounded the need for more proactive alerting and load-testing in lower environments. Without timely notifications, we risk delays in addressing failures that could escalate into more significant problems.
+
+
+## Decision
+We will build out a suite of tools in accordance with industry best practices to monitor our applications. Implementing a comprehensive monitoring and alerting ecosystem will not only help in identifying errors in real-time but also enable us to establish benchmarks based on historical data. This approach will foster a more proactive response strategy, ensuring that potential issues are mitigated before they impact our users or that system owners and system admins are aware of issues that have impacted users.
+
+<p style="text-align:center; margin:0; padding:0;">Cloud Environments Workflow</p>
+
+![Environments](../diagrams/TDP_Environments.png)
+
+### Why Sentry
+Sentry captures unhandled exceptions and incorporates detail context about exceptions including error messages, stack traces, affected URLs and user data information. Such information is essential in demystifying the cause of error.
+
+Additionally, as can be seen in the image below, the following information is available:
+
+- Frequency: shows the frequency detail of error
+- Timeline: when has the error happened in a period
+- Can create a ticket and assign automatically
+- Variables at each step of stack trace. This is very important for debugging
+
+<p style="text-align:center; margin:0; padding:0;">Issues with filter enabled</p>
+
+![Issues with filter enabled](../images/sentry/1.%20Issues%20with%20filter%20enabled.png)
+
+<p style="text-align:center; margin:0;padding:0;">Detail exceptions</p>
+
+![Detail exceptions](../images/sentry/3.%20detail%20about%20exception.png)
+
+<p style="text-align:center; margin:0; padding:0;">Full stack trace of the exceptions</p>
+
+![Full stack trace of the exceptions](../images/sentry/4.%20full%20stack%20trace%20of%20the%20exceptions.png)
+
+
+Performance monitoring in Sentry can greatly enhance the backend application by providing real-time insights into how the TANF app is performing. Sentry tracks various metrics such as response time, database queries, and external API calls. These metrics will help identify performance bottlenecks associated to the backend app.
+
+A unique ability of Sentry is that it links performance issues and groups them together. This gives us the ability to visualize areas that consistently have poor performance. Allowing us to swarm and resolve the most frequent offenders that have the highest impact. Sentry also detects issues with web transactions, database queries, and function regressions (if the duration of function has increased).
+
+### Why Prometheus-Loki-Grafana
+
+Grafana shall provide a visualization dashboard for these various tools which will collect and aggregate performance metrics, system logs, and allow deeper analysis for all aspects of our systems: frontend, proxies, backend, databases, and even networking. Additionally, the development team will seek to hone a proactive alerting system for out-of-threshold issues and errors for improved visibility of system issues.
+
+The storing of system logs will allow more expedient troubleshooting and debugging that is currently out of reach with Cloud.gov's existing Kibana interface for logging. The ability to find and correlate log events is critical to technical analysis of faults, performance degradation, and system's overall health.
+
+By having our monitoring ecosystem take in performance metrics, we will garner performance metrics over time as opposed to simply a live snapshot as is currently provided. This will allow spotting of anomolous or out-of-bounds behaviors such as out of memory, high memory, cpu spikes, and disk thrashing.
+
+Finally, having all of this data in one place will allow technical staff to easily cross-reference given time periods with problematic performance, ongoing issues, or error stacktraces leading to a holistic view of all of our applications both in lower tier development sites and in critical production.
+
+## Consequences
+
+* Increased platform costs for running these tools
+* Time and effort maintaining and configuring these new systems
+* "Noisy" notifications from from out-of-tune alerting
+* Efforts made towards security compliance as these systems have intimate access to our systems and data
+* Learning curve for technical staff
+
+## Notes
+Given the prohibitive costs of self-hosting Sentry in Cloud.gov, we propose using Sentry's Cloud SaaS offering which will alter the [boundary diagram](../../Security-Compliance/diagram.png). The other tools in use (PLG stack and associated), will be self-hosted and maintained by the technical staff both at Raft and OFA.

docs/User-Experience/Research-Syntheses/2020, Summer - Understanding Stakeholders and creating personas.md

@@ -1,6 +1,6 @@
 # Stakeholders and Personas
 
-Last updated for [Issue #804](https://github.com/raft-tech/TANF-app/issues/804)
+Last updated for [Issue #3100](https://github.com/raft-tech/TANF-app/issues/3100)
 
 ---
 
@@ -82,19 +82,19 @@ Some grantees segment data preparation and data transmission into two distinct r
 
 ### OFA analysts
 
-OFA analysts review, analyze, and report on STT’s TANF data that’s submitted through TDP. They also help STTs through the transmission process by offering support and guidance around data quality (including fatal/warning edits). Analysts in the Tribal TANF Division also play a direct role communicating submission status, resubmission requests, and various reports to Tribes. 
+OFA analysts review, analyze, and report on STT’s TANF data that’s submitted through TDP. They also help STTs through the transmission process by offering support and guidance around data quality. 
 
 **Supporting Documentation**
 
 - [Description of the OFA support mailbox](<https://hhsgov.sharepoint.com/:w:/r/sites/TANFDataPortalOFA-UserResearch/Shared%20Documents/User%20Research/OFA%20Resource%20Mailbox%20notes.docx?d=w2542a110040b44c1b82706a54c6ec9f2&csf=1&web=1&e=O5meXQ>) :lock:
 
 ### OFA system admins
 
-OFA admins are members of the TANF Data Division. They oversee the collection and processing of TANF data and ensure OFA analysts have what they need to do their work.
+OFA admins are members of the TANF Data Division. They manage access requests for the TANF data portal, help troubleshoot issues with submission, and broadly ensure that OFA analysts have what they need to do their work.
 
 ### Regional program specialists
 
-Regional program specialists are grantee's primary OFA point of contact. They track TANF trends in their region and provide troubleshooting, advice, and reminders to STTs.
+Regional program specialists are STT's primary OFA point of contact. They track TANF trends in their region and provide troubleshooting, advice, and reminders to STTs.
 
 ### OPRE analysts
 
@@ -109,7 +109,7 @@ respect to TDP, OCIO staff will have access to subsets of the Django Admin exper
 
 ## Personas
 
-Research up to this point has suggested a need for two working personas; an OFA Data Portal User (Grace) and a grantee-side persona. While there's no shortage of variety in roles, responsibilities, and team makeups on both the grantee and federal levels, the "jobs to be done" for each in the TANF Data Portal are shared. A single persona could also have many different combinations of permissions available to them in the data portal. e.g. While the OFA Persona represents all tasks that an OFA user might want to complete, some may only be assigned to some subset of those tasks. 
+Research up to this point has suggested a need for three working personas: an OFA Data Analyst (Grace), an OFA System Admin (Elliott), and an STT data analyst. While these don't reflect the full diversity of  roles, responsibilities, and team makeups on both the grantee and federal levels, the "jobs to be done" for each in the context of TDP have a lot of overlap. A single persona could also have many different combinations of permissions available to them in the data portal. 
 
 ### Grace — OFA Data Portal User
 
@@ -118,11 +118,28 @@ Research up to this point has suggested a need for two working personas; an OFA
 | Role                                | OFA Data Portal User                                         |
 | Age                                 | 35                                                           |
 | Location                            | Washington DC                                                |
-| Work Environment                    | Office                                                       |
-| Goals<br /><br /><br />             | 1. I've identified all states that have or have not transmitted data<br />2. I've identified any outstanding errors in submitted data<br />3. I'm able to contact grantees and facilitate corrections and resubmissions<br />4. I've enabled grantees to fix errors without communication from OFA<br />5. The final TANF data is completely submitted and error free |
-| Pain Points<br /><br /><br /><br /> | 1. Transmission reports can be confusing and aren't actionable enough<br />2. Current errors aren't easy to understand and act to fix, nor easy to explain to grantees<br />3. It can be difficult to identify where an error is coming from in a file<br />4. I see submitted data differently (in different systems) than the grantee submitting it does—The lack of a shared place to view data complicates communication.<br /><br /> |
-| Delights                            | 1. We send out WPR reports far faster than we used to        |
-| Tools                               | OCIO database, scripts to interact with or query the database, email, reporting/analytics software, excel, fTANF.exe |
+| Work Environment                    | Home and Office|
+| Goals<br /><br /><br />             | 1. Identify all states that have or have not transmitted data<br />2. Transform and clean data to generate reports <br />3. Produce reports including: Work participation Rate, Transmission History, Section 1v Section 3, 4 Caseload, Missing Data, Preliminary Quarterly Caseload<br />4. Publish and share reports with STTs, research partners, and colleagues within OFA<br />5. Answer STT questions<br/> 6. Work with the OFA System Admins |
+| Pain Points<br /><br /><br /><br /> | 1. "It can currently take a lot of time to sort out data ready to be reported on from data that's not, and to integrate that into my reporting workflows" <br /> 2. "If data that isn't ready for reporting accidentally makes its way into that workflow then that can cause significant rework and make me have to redo the whole process" <br /> 3. "I don't have a quick way to produce certain reports to communicate to STTs" <br /> 4. "It can be difficult to identify where an error is coming from in a file"                            |
+| Delights                            | 1. "With TDP, I have a lot [of data] once the State submits the data the data gets processed already so that's really good. Now  I don't need to spend too much time with state if the  data wasn't processed." <br /> 2. "Having direct and up to date access to the data in the database" |
+| Tools                               | Django CSV exports, Excel, SAS, email, reporting/analytics software |
+
+
+### Elliott — OFA System Admin
+
+| Grace                               |                                                              |
+| :---------------------------------- | :----------------------------------------------------------- |
+| Role                                | OFA Data Portal User, Viewing and Editing Permissions        |
+| Age                                 | 46                                                           |
+| Location                            | USA, Occasionally In DC Office                              |
+| Work Environment                    | Home and Office, Desktop, and Government Furnished Equipment (GFE) |
+| Goals<br /><br /><br />             | 1. Ensure availability of features and components of the system <br />2. Work with regional OFA staff to confirm who should and shouldn't be granted access to the system<br />3. Access system logs to troubleshoot problems and remediate scans as needed <br />4. Provide access to OCIO (ACF-Tech) staff for security auditing purposes <br />5. Monitoring emails, notifications and answering STT Questions |
+| Pain Points<br /><br /><br /><br /> | 1. "Django doesn't have all the filters I need to efficiently use it for certain admin tasks" <br /> 2. "The system can sometimes log me out with no warning" <br/> 3. Accessing data from cloud.gov-brokered databases is only possible by disconnecting from the HHS network on site and VPN when remote. <br/> 4. The toolstack needed as an administrator can be difficult to use on a Windows GFE <br/> 5. Requesting access to developer tools can take months at a time to get approved                           |
+| Delights                            | 1. "Django has become a tool that provides me a lot of insights about data in the system rather than solely being a user administration tool" <br/> 2. "Automated notifications have been convenient [so that I am aware of what is happening]" <br/> 3. "Being able to use an interface and see what [actions] users did" <br/> 4. "A point and click experience to see how the system is performing...I use the front end the most"|
+| Tools                               | TDP Portal, Django CSV exports, Excel, SAS, email, reporting/analytics software, Microsoft Suite, Python, Axe Dev Tools |
+| Expectations | 1. TDP should help sys admin do things from an interface as much as possible <br /> 2. We need to have multiple ways to troubleshoot and remediate issues |
+
+To view a further in-depth system of the interactions, primary tasks and touchpoints taken by OFA System Admins, please refer to the ecosystem map in the supporting documents below.
 
 ### Awinita — Grantee Data Analyst
 
@@ -141,3 +158,4 @@ Research up to this point has suggested a need for two working personas; an OFA
 
 - [Grantee Attributes](<https://hhsgov.sharepoint.com/:x:/r/sites/TANFDataPortalOFA-UserResearch/_layouts/15/WopiFrame2.aspx?sourcedoc=%7Be83f84f1-71a0-459f-80a4-0d39bc250be9%7D&action=view&cid=f2e656cb-9245-4f02-af6d-2d7e1db6573d>) :lock:
 - [Working Journey Maps & Personas Mural](https://app.mural.co/t/officeoffamilyassistance2744/m/officeoffamilyassistance2744/1608238114372/191c53b8ef538838bc8c179daa238dd5c5dcc9e8) :lock:
+- [Ecosystem Map OFA System Admin](https://app.mural.co/t/raft2792/m/raft2792/1728311825176/74942929a28857c3ef1c401f0d07ccfe1073e882?sender=uf443e00cb4626b8bec157528) :lock:

product-updates/knowledge-center/exporting-complete-data-using-ftanf.html

@@ -345,7 +345,7 @@ <h3 class="usa-process-list__heading">Navigate to the “Utilities” tab, and s
             <a href="../img/complete-data/Step3.jpg"  data-lity data-lity-desc="FTANF.exe opened to the transmission file creation screen with emphasis on the Update Indicator being set to 'Delete'.">
             <img src="../img/complete-data/Step3.jpg" alt="FTANF.exe opened to the transmission file creation screen with emphasis on the Update Indicator being set to 'Delete'.">
           </a>
-            <p>To ensure that your data remains consistent between systems as we’re working to fully replace TDRS with TDP, we ask that you use the Delete value of the Update Indicator in all resubmissions. If you need to confirm that the Delete value has been set correctly, you can look for the “D” value at the end of the Header Record of the file you’re submitting (emphasized below).</p>
+            <p>To ensure that your data remains consistent between FTANF and TDP, please use the Delete value of the Update Indicator in all submissions. To confirm that the Delete value has been set correctly, you can look for the “D” value at the end of the Header Record of the file you’re submitting (emphasized below).</p>
 
             <blockquote style="margin-left: 0px; padding-left: 10px; padding-bottom: 10px; border-color: blue; border-left: solid 3px #005ea2; font-weight: 900;"><span style="font-size: 40px; color: #005ea2; position: relative; right: 10px; padding-left: 10px;">“</span>HEADER20221A01000TAN1 D<span style="font-size: 40px; position: relative; top: 30px; left: 10px; color: #005ea2">”</span></blockquote>
 </ol>