Speed Up Queries: OCI Redis Caching Meets Urban Mobility Data (#489)

* first commit of WMS ID#11612 - Speed Up Queries: OCI Redis Caching Meets Urban Mobility Data first commit of WMS ID#11612 - Speed Up Queries: OCI Redis Caching Meets Urban Mobility Data * fix for Liquid Exception fix for Liquid Exception: Liquid syntax error (line 213): Unknown tag 'block' in oci-cache-with-redis/setup/setup.md * fix done for Liquid Exception: Liquid syntax error fix done for Liquid Exception: Liquid syntax error * WMS:11612 changes done as suggested by reviewer WMS:11612 changes done in lab3 to add more details and images on the tasks as suggested by reviewer * WMS:11612 help email address updated in manifest file WMS:11612 help email address updated in manifest file
oracle-livelabs · Mar 11, 2024 · 84f122d · 84f122d
1 parent b000819
commit 84f122d
Show file tree

Hide file tree

Showing 54 changed files with 992 additions and 0 deletions.
diff --git a/oci-cache-with-redis/data-load/data-load.md b/oci-cache-with-redis/data-load/data-load.md
@@ -0,0 +1,79 @@
+# Load Data into ATP
+
+## Introduction
+
+This lab walks you through the steps to Load taxi trip data into ATP database..
+
+Estimated Time: 15 minutes
+
+### Autonomous Transaction Processing (ATP) 
+Oracle Autonomous Transaction Processing is a fully automated database service optimized to run transactional, analytical, and batch workloads concurrently. To accelerate performance, it’s pre-configured for row format, indexes, and data caching while providing scalability, availability, transparent security, and real-time operational analytics. Application developers and DBAs can rapidly, easily, and cost-effectively develop and deploy applications without sacrificing functionality or atomicity, consistency, isolation, and durability (ACID) properties. Built on Oracle Database and Oracle Exadata, Autonomous Transaction Processing is available on Oracle Cloud Infrastructure (OCI) for serverless or dedicated deployments and on-premises with Exadata Cloud@Customer and OCI Dedicated Region.
+
+### Objectives
+
+In this lab, you will:
+* Download taxi trip data
+* Upload taxi trip data into ATP
+
+### Prerequisites 
+
+This lab assumes you have:
+* An Oracle Cloud account
+* All previous labs successfully completed
+
+
+## Task 1: Download taxi trip data
+
+1. Click [here] (https://www.kaggle.com/c/nyc-taxi-trip-duration/data) and select **test.zip** file and hit download icon.
+
+  ![download taxi trip data](images/download_data_1.png)
+
+2. Once the data is downloaded , unzip and extract the file **test.csv**.
+
+
+## Task 2: Upload data into ATP
+
+1. From OCI console go to Databases > Autonomous transaction Processing. Choose your compartment and select the database **REDISLABATP** you created
+
+  ![select atp database](images/upload_data_1.png)
+
+2. Click on Database actions and select **All Database Actions** and then click on **Data Load**
+
+  ![data load](images/upload_data_2.png)
+
+  ![data load inside](images/upload_data_3.png)
+
+3. Click on **Load Data** and select the file **test.csv** to upload
+
+  ![upload data file](images/upload_data_4.png)
+
+  ![browse file to upload](images/upload_data_5.png)
+
+4. When file is ready to upload , click on **pencil** icon to edit the table name
+
+  ![edit database name](images/upload_data_6.png)
+
+5. Change the table name to **TAXI_TRIPS** and click on close
+
+  ![change database name and close](images/upload_data_7.png)
+
+6. Click **Start** and then click **Run** .
+
+  ![run upload](images/upload_data_8.png)
+
+7. Once load is completed, click on **Done**  ad close the window.
+
+  ![done uploading](images/upload_data_9.png)
+
+  You may now **proceed to the next lab**.
+
+## Learn More
+
+* [About Autonomous transaction processing](https://www.oracle.com/in/autonomous-database/autonomous-transaction-processing/)
+* [About Kaggle datasets](https://www.kaggle.com/datasets)
+
+## Acknowledgements
+* **Author**
+* Pavan Upadhyay, Principal Cloud Engineer, NACI 
+* Saket Bihari, Principal Cloud Engineer, NACI
+* **Last Updated By/Date** - Pavan Upadhyay, Saket Bihari, Feb 2024
diff --git a/oci-cache-with-redis/data-load/images/download_data_1.png b/oci-cache-with-redis/data-load/images/download_data_1.png
diff --git a/oci-cache-with-redis/data-load/images/upload_data_1.png b/oci-cache-with-redis/data-load/images/upload_data_1.png
diff --git a/oci-cache-with-redis/data-load/images/upload_data_2.png b/oci-cache-with-redis/data-load/images/upload_data_2.png
diff --git a/oci-cache-with-redis/data-load/images/upload_data_3.png b/oci-cache-with-redis/data-load/images/upload_data_3.png
diff --git a/oci-cache-with-redis/data-load/images/upload_data_4.png b/oci-cache-with-redis/data-load/images/upload_data_4.png
diff --git a/oci-cache-with-redis/data-load/images/upload_data_5.png b/oci-cache-with-redis/data-load/images/upload_data_5.png
diff --git a/oci-cache-with-redis/data-load/images/upload_data_6.png b/oci-cache-with-redis/data-load/images/upload_data_6.png
diff --git a/oci-cache-with-redis/data-load/images/upload_data_7.png b/oci-cache-with-redis/data-load/images/upload_data_7.png
diff --git a/oci-cache-with-redis/data-load/images/upload_data_8.png b/oci-cache-with-redis/data-load/images/upload_data_8.png
diff --git a/oci-cache-with-redis/data-load/images/upload_data_9.png b/oci-cache-with-redis/data-load/images/upload_data_9.png
diff --git a/oci-cache-with-redis/introduction/introduction.md b/oci-cache-with-redis/introduction/introduction.md
@@ -0,0 +1,45 @@
+# Introduction
+
+## About this Workshop
+
+Welcome to the OCI Redis Cache LiveLab! In this hands-on workshop, we will explore an advanced solution to enhance the performance of a web application that generates pull data stored in Oracle Autonomous Transaction Processing (ATP).
+
+Enquiring records from large datasets and executing complex queries on database can be resource-intensive and time-consuming. To address this challenge, we introduce a powerful solution – leveraging Redis as a caching layer. By caching frequently accessed reports or data in a Redis cluster, we aim to significantly reduce load times, optimize query response times, and ultimately enhance the overall user experience.
+
+Imagine a scenario where you have a web application deployed on a virtual machine (VM). This application retrieves taxi trip data from Oracle ATP and show it online. However, to overcome the potential delays associated with querying ATP directly, we'll implement a Redis cluster to efficiently cache frequently requested data.
+
+Estimated Workshop Time: 2 hours
+
+## Objectives
+
+In this workshop, you will learn how to:
+* Set up and configure a Flask application on a VM
+* Connect the Flask application to Oracle ATP for data retrieval
+* Deploy a Redis cluster to cache frequently accessed data
+* Integrate Redis into the Flask application for efficient caching
+* Test the performance improvements achieved by Redis caching 
+
+## Lab Breakdown
+
+* **Lab 1:** Provision OCI services
+* **Lab 2:** Load Data into ATP
+* **Lab 3:** Setup & Deploy Flask Application in VM
+* **Lab 4:** Run application and test the performance
+
+## Prerequisites 
+
+This lab assumes you have:
+* Access to an Oracle Cloud Infrastructure (OCI) account
+* Basic knowledge of Python and Flask
+* Familiarity with Redis concepts 
+
+## Learn More
+
+* [About OCI Redis](https://docs.oracle.com/en-us/iaas/Content/redis/home.htm)
+* [About OCI ATP](https://docs.oracle.com/en/cloud/paas/atp-cloud/index.html)
+
+## Acknowledgements
+* **Author** 
+* Pavan Upadhyay, Principal Cloud Engineer, NACI 
+* Saket Bihari, Principal Cloud Engineer, NACI
+* **Last Updated By/Date** - Pavan Upadhyay, Saket Bihari, Feb 2024
diff --git a/oci-cache-with-redis/other-livelabs/other-livelabs.md b/oci-cache-with-redis/other-livelabs/other-livelabs.md
@@ -0,0 +1,8 @@
+# Other LiveLabs you might like
+
+
+- [Autonomous Database Dedicated](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/view-workshop?wid=677)
+
+- [Manage and Monitor Autonomous Database](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/view-workshop?wid=553)
+
+- [Scaling and Performance in the Autonomous Database](https://apexapps.oracle.com/pls/apex/dbpm/r/livelabs/view-workshop?wid=608)
diff --git a/oci-cache-with-redis/provision/images/atp.png b/oci-cache-with-redis/provision/images/atp.png
diff --git a/oci-cache-with-redis/provision/images/atp_2.png b/oci-cache-with-redis/provision/images/atp_2.png
diff --git a/oci-cache-with-redis/provision/images/atp_3.png b/oci-cache-with-redis/provision/images/atp_3.png
diff --git a/oci-cache-with-redis/provision/images/atp_4.png b/oci-cache-with-redis/provision/images/atp_4.png
diff --git a/oci-cache-with-redis/provision/images/atp_5.png b/oci-cache-with-redis/provision/images/atp_5.png
diff --git a/oci-cache-with-redis/provision/images/atp_6.png b/oci-cache-with-redis/provision/images/atp_6.png
diff --git a/oci-cache-with-redis/provision/images/cluster_vcn.png b/oci-cache-with-redis/provision/images/cluster_vcn.png
diff --git a/oci-cache-with-redis/provision/images/compute_1.png b/oci-cache-with-redis/provision/images/compute_1.png
diff --git a/oci-cache-with-redis/provision/images/compute_2.png b/oci-cache-with-redis/provision/images/compute_2.png
diff --git a/oci-cache-with-redis/provision/images/compute_3.png b/oci-cache-with-redis/provision/images/compute_3.png
diff --git a/oci-cache-with-redis/provision/images/compute_4.png b/oci-cache-with-redis/provision/images/compute_4.png
diff --git a/oci-cache-with-redis/provision/images/compute_5.png b/oci-cache-with-redis/provision/images/compute_5.png
diff --git a/oci-cache-with-redis/provision/images/create_cluster_pg01.png b/oci-cache-with-redis/provision/images/create_cluster_pg01.png
diff --git a/oci-cache-with-redis/provision/images/create_cluster_pg02.png b/oci-cache-with-redis/provision/images/create_cluster_pg02.png
diff --git a/oci-cache-with-redis/provision/images/create_cluster_pg03.png b/oci-cache-with-redis/provision/images/create_cluster_pg03.png
diff --git a/oci-cache-with-redis/provision/images/create_cluster_pg04.png b/oci-cache-with-redis/provision/images/create_cluster_pg04.png
diff --git a/oci-cache-with-redis/provision/images/create_cluster_pg05.png b/oci-cache-with-redis/provision/images/create_cluster_pg05.png
diff --git a/oci-cache-with-redis/provision/images/redis_cluster.png b/oci-cache-with-redis/provision/images/redis_cluster.png
diff --git a/oci-cache-with-redis/provision/images/vcn.png b/oci-cache-with-redis/provision/images/vcn.png
diff --git a/oci-cache-with-redis/provision/provision.md b/oci-cache-with-redis/provision/provision.md
@@ -0,0 +1,110 @@
+# Provision OCI services
+
+## Introduction
+
+This lab walks you through the steps to provision Redis, ATP and OCI VM instance.
+
+Estimated Time: 45 minutes
+
+### Objectives
+
+In this lab, you will:
+* Provision ATP database
+* Provision Redis cluster
+* Provision VM instance
+
+### Prerequisites
+
+This lab assumes you have:
+* OCI Compartment created
+* IAM policies setup done already
+
+## Task 1: Prepare Redis Cluster
+
+1. Log in to the Oracle Cloud Console as the Cloud Administrator, if you are not already logged in. On the Sign In page, select your tenancy, enter your username and password, and then click Sign In. The Oracle Cloud Console Home page is displayed
+
+2. Follow below link to create VCN
+
+  [Create OCI VCN](https://docs.oracle.com/en/learn/lab_virtual_network/index.html#introduction)
+
+3. Click the Navigation menu and navigate to Databases > Redis > Clusters
+
+  ![navigate to redis cluster](images/redis_cluster.png) 
+
+  Choose the Compartment you created and click **Create cluster**.
+
+4. Provide the cluster name **redis-livelab-cluster** , choose compartment  and click **Next**.
+   ![cluster name](images/create_cluster_pg01.png)
+
+5. Keep the default configuration in Configure nodes tab and click **Next**.
+   ![configure cluster](images/create_cluster_pg02.png)
+
+6. Choose your VCN and Subnet created as part of prerequisite and click **Next**.
+   ![cluster vcn](images/create_cluster_pg03.png)
+
+7. Review the details filled in and click **Create Cluster**.
+   ![review and create cluster](images/create_cluster_pg04.png)
+
+8. Once cluster up and running, copy **OCID** and **Primary endpoint** and keep it ready for flask app configuration in lab-3.
+   ![copy cluster info](images/create_cluster_pg05.png)
+
+## Task 2: Provision ATP Database
+
+1. Click the Navigation menu and navigate to Databases > Autonomous Transaction Processing
+
+  ![navigate to ATP](images/atp.png) 
+
+2. Choose the compartment and click **Create Autonomous Database**
+
+  ![create ATP](images/atp_2.png) 
+
+3. Give Database name as **REDISLABATP**
+
+  ![give ATP a name](images/atp_3.png)
+
+4. Choose a password and keep rest of the fields with default values and click **Create Autonomous Databse**
+
+  ![choose password](images/atp_4.png)
+
+5. Once the ATP is created , click on **Database Connection**.
+
+  ![database connection](images/atp_5.png)
+
+6. Click on **Download Wallet** . After downloading the wallet, unzip it, collect the endpoint from the tnsnames.ora, and set it aside for lab 3.
+
+  ![download wallet](images/atp_6.png)
+
+
+## Task 3: Provision Linux Instance
+
+1. Click the Navigation menu and navigate to Compute > Instances 
+
+  ![navigate to compute](images/compute_1.png)
+
+2. Choose the compartment and click **Create Instance**. 
+
+  ![create instance](images/compute_2.png)
+
+3. Give Instance name **redis-livelab-instance** , choose VCN and public subnet of that VCN , and choose 'Generate a key pair for me' option and leave rest values as default and hit 'Create'
+
+  ![give instance a name](images/compute_3.png)
+
+  ![choose vcn and subnet](images/compute_4.png)
+
+  ![generate and download key pair](images/compute_5.png)
+
+   **Note:** Please save private & public key for login into instance in lab-3
+
+  You may now **proceed to the next lab**.
+
+## Learn More
+
+* [About create Compartment](https://docs.oracle.com/en-us/iaas/Content/Identity/compartments/To_create_a_compartment.htm)
+* [About creating IAM policies for Redis](https://docs.oracle.com/en-us/iaas/Content/redis/permissions.htm)
+* [About creating IAM policies for ATP](https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/adbpolicyreference.htm)
+
+## Acknowledgements
+* **Author** 
+* Pavan Upadhyay, Principal Cloud Engineer, NACI 
+* Saket Bihari, Principal Cloud Engineer, NACI
+* **Last Updated By/Date** - Pavan Upadhyay, Saket Bihari, Feb 2024
diff --git a/oci-cache-with-redis/query/images/check_performance_1.png b/oci-cache-with-redis/query/images/check_performance_1.png
diff --git a/oci-cache-with-redis/query/images/run_app_1.png b/oci-cache-with-redis/query/images/run_app_1.png
diff --git a/oci-cache-with-redis/query/images/run_app_2.png b/oci-cache-with-redis/query/images/run_app_2.png
diff --git a/oci-cache-with-redis/query/query.md b/oci-cache-with-redis/query/query.md
@@ -0,0 +1,62 @@
+# Run application and test the performance
+
+## Introduction
+
+This lab walks you through the steps to run the flask app.
+
+Estimated Lab Time: 15 minutes
+
+### Objectives
+
+In this lab, you will:
+* Run the Flask app
+* Check the log for query performance with ATP/Redis
+
+### Prerequisites 
+
+This lab assumes you have:
+* An Oracle Cloud account
+* All previous labs successfully completed
+
+
+## Task 1: Run the Flask application
+
+1. In VM go to folder **nyctaxi** and run command below.
+
+  Use command below to create the above directory structure.
+  ```
+    <copy>python run.py</copy>
+    ```
+  
+
+2. Open a browser and enter below URL
+
+     ```
+    <copy>http://&ltpublic ip of VM&gt:5000</copy>
+    ```
+
+3. Enter start and end date and click on **search**. 
+    
+    ![search data](images/run_app_1.png)
+
+4. Once the result are displayed, click on **New Search** and enter the same date range and **search** again
+    
+    ![new search](images/run_app_2.png)
+
+**Note** - New search will always get data from ADW and repeated search will get data from Redis
+
+## Task 2: Check the performance of the query from logs
+
+1. Go to the terminal where app is running and see the console log for print statements **Time taken to retrieve data...**. You will see the significant difference between ATP & Redis query performance
+    
+    ![log](images/check_performance_1.png)
+
+## Learn More
+
+* [About OCI](https://www.oracle.com/in/cloud/)
+
+## Acknowledgements
+* **Author** 
+* Pavan Upadhyay, Principal Cloud Engineer, NACI 
+* Saket Bihari, Principal Cloud Engineer, NACI
+* **Last Updated By/Date** - Pavan Upadhyay, Saket Bihari, Feb 2024
diff --git a/oci-cache-with-redis/setup/images/deploy_app_1.png b/oci-cache-with-redis/setup/images/deploy_app_1.png
diff --git a/oci-cache-with-redis/setup/images/installation_1.png b/oci-cache-with-redis/setup/images/installation_1.png
diff --git a/oci-cache-with-redis/setup/images/login_vm_1.png b/oci-cache-with-redis/setup/images/login_vm_1.png
diff --git a/oci-cache-with-redis/setup/images/login_vm_2.png b/oci-cache-with-redis/setup/images/login_vm_2.png
diff --git a/oci-cache-with-redis/setup/images/login_vm_3.png b/oci-cache-with-redis/setup/images/login_vm_3.png
diff --git a/oci-cache-with-redis/setup/images/verify_config.png b/oci-cache-with-redis/setup/images/verify_config.png
diff --git a/oci-cache-with-redis/setup/images/verify_init.png b/oci-cache-with-redis/setup/images/verify_init.png
diff --git a/oci-cache-with-redis/setup/images/verify_models.png b/oci-cache-with-redis/setup/images/verify_models.png
diff --git a/oci-cache-with-redis/setup/images/verify_routes.png b/oci-cache-with-redis/setup/images/verify_routes.png
diff --git a/oci-cache-with-redis/setup/images/verify_run.png b/oci-cache-with-redis/setup/images/verify_run.png