GITBOOK-1018: ODK Importer - SR

SUMMARY.md

@@ -271,6 +271,9 @@
     * [SPAR Update for offline enumerations](social-registry/features/spar-update-for-offline-enumerations.md)
     * [Self Service Registration Portal](social-registry/features/self-service-registration-portal.md)
     * [Unique Reference ID](social-registry/features/unique-reference-id.md)
+    * [ODK Importer](social-registry/features/odk-importer/README.md)
+      * [📔 User Guide](social-registry/features/odk-importer/user-guide/README.md)
+        * [📔 Configure and Import ODK Form](social-registry/features/odk-importer/user-guide/configure-and-import-odk-form.md)
   * [Versions](social-registry/versions.md)
   * [Deployment](social-registry/deployment/README.md)
     * [Domain names and Certificates](social-registry/deployment/domain-names-and-certificates.md)
@@ -279,10 +282,10 @@
     * [Technology Stack](social-registry/developer-zone/technology-stack.md)
     * [Repositories](social-registry/developer-zone/repositories.md)
     * [Developer Install](social-registry/developer-zone/developer-install/README.md)
-      * [📘 Developer Install of OpenG2P Package on Linux](social-registry/developer-zone/developer-install/developer-install-of-openg2p-package-on-linux.md)
+      * [📘 Developer Install of OpenG2P Package on Linux](social-registry/developer-zone/developer-install-of-openg2p-package-on-linux.md)
     * [Packaging](social-registry/developer-zone/packaging/README.md)
-      * [📘 Docker Packaging Guide](social-registry/developer-zone/packaging/docker-packaging-guide.md)
-      * [📘 Helm Packaging Guide](social-registry/developer-zone/packaging/helm-packaging-guide.md)
+      * [📘 Docker Packaging Guide](social-registry/developer-zone/docker-packaging-guide.md)
+      * [📘 Helm Packaging Guide](social-registry/developer-zone/helm-packaging-guide.md)
     * [Odoo Modules](social-registry/developer-zone/odoo-modules/README.md)
       * [ODK App User Mapping](social-registry/developer-zone/odoo-modules/odk-app-user-mapping.md)
 * [SPAR](spar/README.md)

social-registry/developer-zone/developer-install/developer-install-of-openg2p-package-on-linux.md → social-registry/developer-zone/developer-install-of-openg2p-package-on-linux.md

@@ -244,7 +244,7 @@ db_password = your_database_password
 ## Installation of OpenG2P package
 
 1. Create a `custom-addons` folder inside the `odoo` folder to keep all the extra modules.
-2.  Clone all the [OpenG2P modules](../../../pbms/development/repositories/).
+2.  Clone all the [OpenG2P modules](../../pbms/development/repositories/).
 
     ```bash
     git clone <repo_url>

social-registry/developer-zone/packaging/helm-packaging-guide.md → social-registry/developer-zone/helm-packaging-guide.md

@@ -2,10 +2,10 @@
 
 ## Concepts
 
-Social Registry and all its dependencies are installed using a single packaged [Helm chart](https://github.com/OpenG2P/openg2p-social-registry-deployment/tree/develop/charts).  The contents of the package may be found in the [Docker package files](https://github.com/OpenG2P/openg2p-packaging/tree/main/packaging/packages/social-registry).  Learn more on Helm charts, versioning and packaging [here](../../../deployment/helm-charts.md#helm-chart-versions).
+Social Registry and all its dependencies are installed using a single packaged [Helm chart](https://github.com/OpenG2P/openg2p-social-registry-deployment/tree/develop/charts).  The contents of the package may be found in the [Docker package files](https://github.com/OpenG2P/openg2p-packaging/tree/main/packaging/packages/social-registry).  Learn more on Helm charts, versioning and packaging [here](../../deployment/helm-charts.md#helm-chart-versions).
 
 {% hint style="info" %}
-\* The chart has been split into parts to address the Kubernetes ETCD limitation. [Learn more >>](../../../deployment/helm-charts.md#helm-chart-size-limitation)
+\* The chart has been split into parts to address the Kubernetes ETCD limitation. [Learn more >>](../../deployment/helm-charts.md#helm-chart-size-limitation)
 {% endhint %}
 
 Helm package is the highest level of package that is offered for module installation.  The Helm package contains all dependencies and is intended to be installed "single click" from Rancher or the command line.
@@ -29,7 +29,7 @@ The charts are located in [this](https://github.com/OpenG2P/openg2p-social-regis
 4. Check in the changes on this branch
 5. Create a tag following tagging conventions. E.g. `v1.3.1`.  
 6. The Github workflow action to package Helm and push on `openg2p-helm` repo's `gh-pages` branch should be triggered automatically and the chart published.
-7. Update the [Versions](../../versions.md) page on this documentation.
+7. Update the [Versions](../versions.md) page on this documentation.
 
 {% hint style="warning" %}
 The Github workflow triggers only if the branch already exists and changes are applied on the branch. So create the branch first on Github directly and push the Helm changes.

social-registry/features/odk-importer/README.md

@@ -0,0 +1,73 @@
+---
+description: Work in Progress
+---
+
+# ODK Importer
+
+The Social Registry (SR) gathers registrants'/applicants' data through various mechanisms, one of which includes ODK forms to import data housed within ODK Central. 
+
+OpenG2P has built an ODK Importer module. It is specifically designed to tailor, configure the required field based on the ODK (Open Data Kit) form, and import the beneficiaries' relevant ODK form into SR module. ODK forms are customised in compliance with project requirements using the JSON Formatter. The benefits of using ODK Importer are:
+
+* Streamlines data transfer from ODK forms directly into SR modules.
+* Eliminates manual data entry, improves data accuracy, and efficiency in data management.
+* Provides data import for automatic updates using the automated scheduler.
+
+## Feature and functionality
+
+<table><thead><tr><th width="213">Feature </th><th>Functionality</th></tr></thead><tbody><tr><td><strong>Data Import</strong></td><td><ul><li>Simplifies the process of entering data from the ODK forms' field into individual/group registries of the SR module, ensuring accuracy and effectiveness in data management.</li></ul></td></tr><tr><td><strong>Test Connection</strong></td><td><ul><li>Verifies the established connection between the ODK Importer and ODK Central.</li><li>Ensures that the ODK Importer can effectively communicate with ODK Central, the server hosting the ODK forms. </li><li>Enables seamless data transfer between the two systems.</li></ul></td></tr><tr><td><strong>Import Records</strong></td><td><ul><li>Tracks the progress of data import and identifies any errors that may occur during the process  </li><li>Provides users with real-time updates on the status of their data import, allowing them to quickly address any issues and ensure a smooth import process</li></ul><p>Note:</p><p>This feature is used when you want to retrieve data in bulk.</p></td></tr><tr><td><strong>Fetch Records</strong></td><td><ul><li>This feature enables users to retrieve a specific record from the ODK system using the unique ODK Instance ID. It works similarly to the standard import record process but offers more precision by allowing users to target a single record based on its unique identifier. </li><li>This is particularly useful when you need to retrieve specific record without importing the complete dataset.</li></ul><p>Note: </p><p>Make sure you have completed the Configuration procedure to use this feature. To learn more on this feature, refer Import Specific ODK Form.</p></td></tr><tr><td><strong>Automated Data Processing</strong></td><td><ul><li>Removes the risk of manual error, ensuring higher accuracy in data records.</li><li>Streamlines task, saves time and resources while enhancing overall operational efficiency.</li><li>Increases total operational efficiency, streamlines tasks, saves time and resources.</li></ul></td></tr><tr><td><strong>Automated Scheduler</strong></td><td><ul><li>Allows smooth updates that do not require operator intervention. This capability enables rapid and regular changes to data sets, improving data currency and relevancy</li><li>Automates the import process, improves workflow efficiency, and avoids delay in processing</li></ul></td></tr></tbody></table>
+
+## Process workflow
+
+{% embed url="https://miro.com/app/board/uXjVKcYGOyg=/?share_link_id=96965246843" %}
+
+## **Process & Data Flow**
+
+1. **User initiates import:** User selects the desired ODK form and configures import options.
+2. **Data Mapping:** User establishes a mapping between corresponding fields in the ODK form using JSON-formatter (PYJQ) and the OpenG2P system.
+3. **Connection & Data Retrieval:** Module connects to ODK Central and retrieves data based on the selected form and import options.
+4. **Data Import:** Extracted data is imported into the designated fields within the OpenG2P system.
+5. **Import Status Update:** User receives notification of successful import completion or encounters any errors.
+
+## Concepts
+
+1. **ODK Form**: This is the source of data collection through the ODK. ODK forms are used to collect various types of data, such as beneficiary information in a structured format.
+2. **Mapping**: Mapping defines the correspondence between fields in the ODK form and the OpenG2P system. It ensures that data collected in the ODK form is accurately mapped to the corresponding fields within the OpenG2P system.
+
+## ODK import
+
+After the ODK configuration, the fields/parameters in the ODK form are mapped to the relevant fields in SR module using the JSON formatter. Navigate to the path Social Registry -> ODK -> ODK Import -> New. To learn more on ODK Import, refer to [Configure and Import ODK Form](user-guide/configure-and-import-odk-form.md).
+
+Here is the sample ODK import configuration form.
+
+<figure><img src="../../../.gitbook/assets/odk-import-form (1).png" alt=""><figcaption></figcaption></figure>
+
+## Configuration
+
+After the computation of required fields in SR, the SR module built on Odoo platform needs to be integrated with ODK Central to facilitate data collection and integration. SR module must be set up to automatically pull data from the ODK forms located in the ODK Central. Navigate to the path Social Registry -> ODK -> Configuration -> New.  To learn more on Configuration, refer to [Configure and Import ODK Form](user-guide/configure-and-import-odk-form.md).
+
+Here is the sample _**Configuration**_ screen.
+
+<figure><img src="../../../.gitbook/assets/odk-configuration.png" alt=""><figcaption></figcaption></figure>
+
+## **Source code**
+
+[https://github.com/OpenG2P/openg2p-importers/tree/17.0-develop/g2p\_odk\_importer](https://github.com/OpenG2P/openg2p-importers/tree/17.0-develop/g2p\_odk\_importer)
+
+## **Technical concepts**
+
+* Utilizes APIs to communicate with ODK Central for data retrieval.
+* Implements error handling mechanisms for robust import processes.
+
+## API docs
+
+Below is the documentation link for the APIs used in the ODK configuration between ODK Central and SR.
+
+* [https://docs.getodk.org/central-api-accounts-and-users/](https://docs.getodk.org/central-api-accounts-and-users/)
+
+## Installation and deployment
+
+Odoo installation
+
+## Related user guides
+
+[Configure and Import ODK Form](user-guide/configure-and-import-odk-form.md).

social-registry/features/odk-importer/user-guide/README.md

@@ -0,0 +1,2 @@
+# 📔 User Guide
+

social-registry/features/odk-importer/user-guide/configure-and-import-odk-form.md

@@ -0,0 +1,132 @@
+---
+layout:
+  title:
+    visible: true
+  description:
+    visible: false
+  tableOfContents:
+    visible: true
+  outline:
+    visible: true
+  pagination:
+    visible: true
+---
+
+# 📔 Configure and Import ODK Form
+
+This document provides step-by-step instructions for configuring the ODK central into the Social Registry (SR) and importing the ODK form. After importing the ODK form, only the fields and their values customised using the JSON formatter are included in the individual/group registries of the SR.
+
+Note:
+
+Every ODK form must have its own ODK importer.
+
+## Prerequisites
+
+* A user must have an Administrator role to access ODK Central in SR.
+* A user must know the pyjq syntax to customise the value in the JSON Formatter fields.
+
+## Procedure
+
+1. Click the main icon ![](../../../../.gitbook/assets/menu-icon.png) and select ODK.
+
+<figure><img src="../../../../.gitbook/assets/sr-odk.png" alt=""><figcaption></figcaption></figure>
+
+_**ODK**_ screen is displayed. You can find two tabs.
+
+* [ODK Import](configure-and-import-odk-form.md#odk-import)
+* [Configuration](configure-and-import-odk-form.md#odk-configuration)
+
+### ODK Import
+
+By default, ODK -> ODK Import page is displayed.
+
+<figure><img src="../../../../.gitbook/assets/odk-import-sr.png" alt=""><figcaption></figcaption></figure>
+
+2. Click the **New** button.
+
+_**ODK Import New**_ screen is displayed.
+
+<figure><img src="../../../../.gitbook/assets/odk-import-new-sr.png" alt=""><figcaption></figcaption></figure>
+
+The fields and their descriptions are given below.
+
+3. Customize the ODK Importer based on the ODK form fields
+
+<table><thead><tr><th width="226">Parameters</th><th>Description</th></tr></thead><tbody><tr><td><strong>ODK CONFIGURATION</strong></td><td></td></tr><tr><td><strong>ODK  Config</strong></td><td>The name of the ODK configuration</td></tr><tr><td><strong>TARGET SETTINGS</strong></td><td></td></tr><tr><td><strong>Target Registry</strong></td><td><p>Specifies whether imported records should be associated with individual or group registries within the SR.</p><p>Select the appropriate value from the drop-down. The valid values are:</p><ul><li>Individual</li><li>Groups</li></ul></td></tr><tr><td><strong>JSON Formatter</strong></td><td>Enables users to define a custom JSON formatter for transforming retrieved data before importing it into the SR. Refer to relevant documentation for details on utilizing the JSON formatter using <a href="https://pypi.org/project/pyjq/">PYJQ library</a>.</td></tr><tr><td><strong>TIME INTERVAL</strong></td><td></td></tr><tr><td><strong>Interval in Hours (Optional)</strong></td><td>Defines the interval (in hours) at which the import process should be automatically triggered. This allows for scheduled data updates.</td></tr><tr><td><a href="configure-and-import-odk-form.md#test-connection"><strong>Test Connection</strong></a></td><td><ul><li>Verifies the established connection between the ODK Importer and ODK Central.</li><li>Ensures that the ODK Importer can effectively communicate with ODK Central, the server hosting the ODK forms.</li><li>Enables seamless data transfer between the two systems</li></ul></td></tr><tr><td><a href="configure-and-import-odk-form.md#import-records"><strong>Import Records</strong></a></td><td><ul><li>Tracks the progress of data import and identifies any errors that may occur during the process.</li><li>Provides users with real-time updates on the status of their data import, allowing them to quickly address any issues and ensure a smooth import process.</li></ul></td></tr><tr><td><strong>Start/Stop</strong></td><td><p> <em><strong>Start</strong></em> executes the scheduled job to run periodically at fixed times, dates, or intervals.</p><p><em><strong>Stop</strong></em> terminates the scheduled job.</p></td></tr></tbody></table>
+
+<table><thead><tr><th width="108">Icon</th><th>Click to</th></tr></thead><tbody><tr><td><img src="../../../../.gitbook/assets/Actions.png" alt="" data-size="original"></td><td><p>Select the appropriate value. The available values are: </p><ul><li>Archive</li><li>Duplicate</li><li>Change Password</li><li>Disable two-factor authentication</li><li>Send Password Reset Instructions</li><li>Privacy Lookup</li></ul></td></tr><tr><td><img src="../../../../.gitbook/assets/icon-save-manually.png" alt="" data-size="original"></td><td>Save manually the individual data and exit from the screen.</td></tr><tr><td><img src="../../../../.gitbook/assets/discard-changes-icon.png" alt="" data-size="original"></td><td>Discard changes and exit from the screen.</td></tr></tbody></table>
+
+<figure><img src="../../../../.gitbook/assets/odk-import-form.png" alt=""><figcaption></figcaption></figure>
+
+## Configuration
+
+4. Click the tab _**Configuration**_ in the menu bar.
+
+_**Configuration**_ screen is displayed.
+
+<figure><img src="../../../../.gitbook/assets/configuration-sr.png" alt=""><figcaption></figcaption></figure>
+
+5. Click the _**New**_ button.
+
+_**Configuration New**_ screen is displayed.
+
+<figure><img src="../../../../.gitbook/assets/configuration-new-sr.png" alt=""><figcaption></figcaption></figure>
+
+The fields and their descriptions are given below.
+
+| Field                   | Description                                                            |
+| ----------------------- | ---------------------------------------------------------------------- |
+| _**ODK Configuration**_ |                                                                        |
+| Name                    | Enter the name for the ODK importer.                                   |
+| Base URL                | Enter the URL of the ODK Central.                                      |
+| Username                | Enter the username which is used to log in ODK Central                 |
+| Password                | Enter the password which is used to login ODK Central                  |
+| _**Project details**_   |                                                                        |
+| Project                 | <p>Enter the project number. </p><p>For example, 3</p>                 |
+| Form ID                 | <p>Enter the ID of the form.</p><p>For example, Safety-Net-Program</p> |
+
+**Sample ODK form URL**
+
+<figure><img src="../../../../.gitbook/assets/odkformurl-sr.png" alt=""><figcaption></figcaption></figure>
+
+### Test Connection
+
+_**Test Connection**_ feature establishes the connection between _**ODK Central**_ and _**SR**_.
+
+7. Click the _**Test Connection**_ button.
+
+If the _**Test Connection**_ feature connects the _**ODK Central**_ and _**SR**_ successfully, a successful message pop ups.
+
+<figure><img src="../../../../.gitbook/assets/test-connection-sr.png" alt=""><figcaption></figcaption></figure>
+
+If the _**Test Connection**_ feature unable to connect the _**ODK Central**_ and _**SR**_, an error  message pop ups.&#x20;
+
+<figure><img src="../../../../.gitbook/assets/test-connection-fail-sr.png" alt=""><figcaption></figcaption></figure>
+
+8. Click the _**OK**_ button to exit the dialog box.
+
+### Import Records
+
+_**Import Records**_ feature imports and stores the records in SR.
+
+9. Click the _**Import Records**_ button.
+
+If the ODK form is imported successfully, a success message pops up.
+
+<figure><img src="../../../../.gitbook/assets/import-record-success.png" alt=""><figcaption></figcaption></figure>
+
+If no new ODK forms are submitted, ODK form records will not be imported.
+
+<figure><img src="../../../../.gitbook/assets/import-record-fail.png" alt=""><figcaption></figcaption></figure>
+
+You can find the configuration details in the _**ODK Configuration**_ screen and imported ODK forms' status in the _**ODK Import**_ screen.
+
+This completes the process of importing an ODK form from ODK Central into the SR.
+
+### JSON Formatter
+
+Configure the required fields based on the ODK form field using pyjq JSON formatter.&#x20;
+
+**Sample JSON Formatter**
+
+<figure><img src="../../../../.gitbook/assets/JSON-formatter-1.png" alt=""><figcaption><p>JSON Formatter</p></figcaption></figure>