From ae8057fc64589a63b9d484e50fbfac26912dbb7e Mon Sep 17 00:00:00 2001
From: Cristhian Garcia <crisgarta8@gmail.com>
Date: Wed, 10 Apr 2024 12:42:13 -0500
Subject: [PATCH] chore: update readme with settings

---
 README.rst                                   | 140 ++++++++++++++++++-
 docs/how-tos/load_test_tracking_pipeline.rst |   2 +
 2 files changed, 137 insertions(+), 5 deletions(-)

diff --git a/README.rst b/README.rst
index 4f9b2f7..cf5f21c 100644
--- a/README.rst
+++ b/README.rst
@@ -4,8 +4,55 @@ Platform Plugin Aspects
 Purpose
 *******
 
-This repository holds various Aspects related plugins for edx-platform.
+This repository holds various Aspects plugins for the Open edX platform.
 
+Sinks
+*****
+
+Sinks are components that listen for events and store them in ClickHouse. The
+events are emitted by the Open edX platform via `Open edX events`_ or Django signals.
+
+Available Sinks
+===============
+
+- `CourseOverviewSink` - Listens for the `COURSE_PUBLISHED` event and stores the
+  course structure data through `XBlockSink` in ClickHouse.
+- `ExternalIdSink` - Listens for the `post_save` Django signal on the `ExternalId`
+  model and stores the external id data in ClickHouse.
+- `UserProfile` - Listens for the `post_save` Django signal on the `UserProfile`
+  model and stores the user profile data in ClickHouse.
+- `UserRetirementSink` - Listen for the `USER_RETIRE_LMS_MISC` Django signal and
+  remove the user PII information from ClickHouse.
+
+Commands
+========
+
+In addition to being an event listener, this package provides the following commands:
+
+- `dump_data_to_clickhouse` - This command allows bulk export of the data from the Sinks.
+  Allows bootstrapping a new data platform or backfilling lost or missing data.
+
+    ``python manage.py cms dump_data_to_clickhouse``
+
+- `load_test_tracking_events` - This command allows loading test tracking events into
+  ClickHouse. This is useful for testing the ClickHouse connection to measure the performance of the
+  different data pipelines such as Vector, Event Bus (Redis and Kafka), and Celery.
+
+  Do not use this command in production as it will generate a large amount of data
+  and will slow down the system.
+
+    ``python manage.py cms load_test_tracking_events``
+
+- `monitor_load_test_tracking` - Monitors the load test tracking script and saves
+  output for later analysis.
+
+    ``python manage.py cms monitor_load_test_tracking``
+
+Instructor Dashboard Integration
+================================
+
+Dashboards from `Aspects`_ are integrated into the Instructor Dashboard via `Superset Embedded SDK`_.
+See `Configuration`_ for more details.
 
 Getting Started with Development
 ********************************
@@ -15,12 +62,88 @@ Please see the Open edX documentation for `guidance on Python development <https
 Deploying
 *********
 
-TODO: How can a new user go about deploying this component? Is it just a few
-commands? Is there a larger how-to that should be linked here?
+The `Platform Plugin Aspects` component is a django plugin which doesn't
+need independent deployment. Therefore, its setup is reasonably straightforward.
+First, it needs to be added to your service requirements, and then it will be
+installed alongside requirements of the service.
+
+Configuration
+*************
+
+Instructor Dashboard Configuration
+==================================
+
+The Instructor Dashboard integration uses the `Open edX Filters`_. To learn more about
+the filters, see the `Open edX Filters`_ documentation. Make sure to configure the
+superset pipeline into the filter as follows:
 
-PLACEHOLDER: For details on how to deploy this component, see the `deployment how-to`_
+    .. code-block:: python
+
+      OPEN_EDX_FILTERS_CONFIG = {
+        "org.openedx.learning.instructor.dashboard.render.started.v1": {
+          "fail_silently": False,
+          "pipeline": [
+            "platform_plugin_superset.extensions.filters.AddSupersetTab",
+          ]
+        },
+      }
+
+- `SUPERSET_CONFIG` - This setting is used to configure the Superset Embedded SDK.
+  The configuration is a dictionary that contains the following keys:
+
+    - `internal_service_url` - The URL of the Superset instance (useful in development, omit in production).
+    - `service_url` - The URL of the Superset instance.
+    - `username` - The username of the Superset user.
+    - `password` - The password of the Superset user.
+
+- `ASPECTS_INSTRUCTOR_DASHBOARDS` - This setting is used to configure the dashboards
+  that will be displayed in the Instructor Dashboard. The configuration is a list of
+  dictionaries that contains the following keys:
+
+    - `name` - The name of the dashboard.
+    - `slug` - The slug of the dashboard.
+    - `uuid` - The UUID of the dashboard.
+    - `allow_translations` - A boolean value that determines if the dashboard
+      is translated in `Aspects`_.
+
+- `SUPERSET_EXTRA_FILTERS_FORMAT` - This setting is used to configure the extra filters
+  that will be applied to the dashboards. The configuration is a list of strings that
+  can be formatted with the following variables:
+
+    - `user` - The user object.
+    - `course` - The course object.
+
+- `SUPERSET_DASHBOARD_LOCALES` - This setting is used to configure the available locales
+  for the dashboards. The configuration is a list of supported locales by `Aspects`_.
+
+Event Sink Configuration
+========================
+
+- `EVENT_SINK_CLICKHOUSE_BACKEND_CONFIG` - This setting is used to configure the ClickHouse
+  connection. The configuration is a dictionary that contains the following keys:
+
+    - `url` - The host of the ClickHouse instance.
+    - `database` - The database name.
+    - `username` - The username of the ClickHouse user.
+    - `password` - The password of the ClickHouse user.
+    - `timeout_secs` - The timeout in seconds for the ClickHouse connection.
+
+- `EVENT_SINK_CLICKHOUSE_PII_MODELS` - This setting is used to configure the models that
+  contain PII information. The configuration is a list of strings that contain the
+  table names where the PII information is stored.
+
+- `EVENT_SINK_CLICKHOUSE_MODEL_CONFIG` - This setting is used to provide compatibility
+  with multiple Open edX models. The configuration is a dictionary that contains the
+  following a key per model that contains a dictionary with the following keys:
+
+    - `module` - The module path of the model.
+    - `model` - The model class name.
+
+Event Sinks are disabled by default. To enable them, you need to enable the following
+waffle flag: `event_sink_clickhouse.{{model_name}}.enabled` where model name is the name
+of the model that you want to enable. Or, you can enable them via settings by setting
+`EVENT_SINK_CLICKHOUSE_{{model_name}}_ENABLED` to `True`.
 
-.. _deployment how-to: https://docs.openedx.org/projects/platform-plugin-aspects/how-tos/how-to-deploy-this-component.html
 
 Getting Help
 ************
@@ -96,3 +219,10 @@ Reporting Security Issues
 *************************
 
 Please do not report security issues in public. Please email security@openedx.org.
+
+.. _Open edX events: https://github.com/openedx/openedx-events
+.. _Edx Platform: https://github.com/openedx/edx-platform
+.. _ClickHouse: https://clickhouse.com
+.. _Aspects: https://docs.openedx.org/projects/openedx-aspects/en/latest/index.html
+.. _Superset Embedded SDK: https://www.npmjs.com/package/@superset-ui/embedded-sdk
+.. _Open edX Filters: https://docs.openedx.org/projects/openedx-filters/en/latest/
diff --git a/docs/how-tos/load_test_tracking_pipeline.rst b/docs/how-tos/load_test_tracking_pipeline.rst
index 0868a40..9ead2fd 100644
--- a/docs/how-tos/load_test_tracking_pipeline.rst
+++ b/docs/how-tos/load_test_tracking_pipeline.rst
@@ -1,3 +1,5 @@
+.. _load-test-tracking-event-pipelines:
+
 Load Testing Tracking Event Pipelines
 #####################################