Amazon ElastiCache Caching for RDS Solutions Guidance

Author: rlunar

guidance/README.md

@@ -29,14 +29,14 @@ You are responsible for the cost of the AWS services used while running this Gui
 As of 03/28/2024, the cost for running this guidance with the default settings using AWS ElastiCache instance type cache.t2.x.small utilizing 1 primary, in the US East (N. Virginia) for on-demand pricing is approximately $24.82 USD total per month. AWS The RDS database cost is estimated at $14.71. However, cost will greatly depend on the, instance size, and RDS licensing model selected. Reserved instance pricing will reduce cost for both RDS and ElastiCache. ElastiCache is also available in a serverless offering where pay-per-consumption cost model is applicable. 
 
 | Service               | Assumptions                                       | Estimated Cost Per Month |
-| --------------------- | ------------------------------------------------- | -------------            | 
-| Amazon ElastiCache    | 2 Instance (cache.t2.small) used for 730 hours	| $24.82                   |
-| Amazon RDS MySQL      | 1 Instance (db.t3.micro) used for 730 hours       | $14.71                   |
+| --------------------- | ------------------------------------------------- | ------------- | 
+| Amazon ElastiCache    | 2 Instance (cache.t2.small) used for 730 hours	| $24.82 |
+| Amazon RDS MySQL      | 1 Instance (db.t3.micro) used for 730 hours       | $14.71 |
 
 
 ## Prerequisites
 
-This guidance is targeted towards those familiar with the AWS RDS Service. The users are expected to have a basic understanding of AWS RDS service and database access patterns. It guides users how to utilize AWS ElastiCache in addition to their existing relational database. Effectively paring your database with a caching service. It should be run in US East N.Virginia region.
+This guidance is targeted towards those familiar with the AWS RDS Service. The users are expected to have a basic understanding of AWS RDS service and database access patterns. It guides users how to utilize AWS ElastiCache in addition to their existing relational database. Effectively paring your database with a caching service. It should be run in US East N.Virginia region. This guidance is not intended for production workloads. For example, recommended security configurations are not included. For production systems it is strongly recommended that data should be encrypted both in transit and at rest.
 
 
 ### Operating System
@@ -76,9 +76,9 @@ All regions where AWS RDS MySQL and AWS ElastiCache are offered.
 3. Clone the repository by executing ```git clone <repo name> ```
 4. Change directory to the guidance directory ```cd guidance```
 5. Execute the setup_host script ```./setup_host.sh```
-6. Log in to the same instance from a separate session and navigate to the same directory and execute ```./setup_jupyter.sh``` script.
-7. In your computer browser enter the EC2's public IP address and port for example ```http://1.2.3.4:8888`` Note that this is not a secured service. 
-8. Enter the preset password for your Jupyter notebook "test123"
+6. Log in to the same instance from a separate session and navigate to the same directory and execute ```./setup_jupyter.sh``` script. Enter the initial password. Commit this to memory as you will have to enter it once the notebook is running.
+7. In your computer browser enter the EC2's public IP address and port for example ```http://1.2.3.4:8888`` Note: that jupyter configuration is not a secured as it is running the http (no s) protocol. In order to secure jupyter notebook please configure https protocol as documented here: https://jupyter-notebook.readthedocs.io/en/6.2.0/public_server.html#running-a-public-notebook-server 
+8. Enter the password for your Jupyter notebook. (The password entered at step 6)
 9. In your first session edit the .env file and update it with your database and ElastiCache related information.
 10. Source the .env file ```source .env``` to export the parameters.
 
@@ -88,26 +88,20 @@ It is not part of this guidance to install and configure client applications for
 
 ## Running the Guidance
 
-* Execute the `scenario01.py` script. This workload accesses the database only and captures command level performance data in a logfile. In the directory where you executed the `setup_host.sh` and the Python virtual environment is activated, the first connection, execute: 
-```bash
-(.venv) [ec2-user]$ python scenario01.py --users 10 --queries 1000 --read_rate 80
-```
-
+* Execute the scenario01.py script. This workload accesses the database only and captures command level performance data in a logfile. In the directory where you executed the setup_host.sh and the Python virtual environment is activated, the first connection, execute: ```python scenario01.py --users 10 --queries 1000 --read_rate 80```
 * If deployment was correct  you should see a response similar to this. (small sample execution)
-
-```bash  
+  
 (.venv) [ec2-user]$ python scenario01.py --users 1 --queries 10 --read_rate 80
 Reads: 8
 Writes: 2
 Logfile located here: logs/scenario01_139007_mwae8c4k.json
-```
-* Open the Jupyter notebook `plot_results_db_only.ipynb` file and update the logfile name in the second cell. For example ```log_pattern = 'scenario01_139007_mwae8c4k.json```
+
+* Open the Jupyter notebook plot_results_db_only.ipynb file and update the logfile name in the second cell. For example ```log_pattern = 'scenario01_139007_mwae8c4k.json```
 
 * From the run option select run all cells. The output of the last cell will show both the number of executions per second and the average response time. 
 
 * To compare the performance boost provided by ElastiCache repeat the above steps but use the scenario02.py script. For example execute ```python scenario02.py --users 1 --queries 10 --read_rate 80``` The output should be similar.
 
-```bash
 (.venv) [ec2-user]$ python scenario02.py --users 1 --queries 10 --read_rate 80
 Connected to Database
 Connected to ElastiCache
@@ -116,9 +110,8 @@ Writes: 0
 Cache hits: 10
 Cache misses: 0
 Logfile located here: logs/scenario02_176908_0y2qr55f.json
-```
 
-* Open the Jupyter notebook `plot_results_db_and_cache.ipynb` file and update the logfile name in the second cell. For example ```log_pattern = 'scenario02_176908_0y2qr55f.json```
+* Open the Jupyter notebook plot_results_db_and_cache.ipynb file and update the logfile name in the second cell. For example ```log_pattern = 'scenario02_176908_0y2qr55f.json```
 
 Then select run all cells to plot the performance of the second scenario. Note that a small execution may not be sufficient to demonstrate the performance advantage of adding a cache. 
 
@@ -148,3 +141,4 @@ For feedback please access the github page <here>
 Include a legal disclaimer
 
 *Customers are responsible for making their own independent assessment of the information in this Guidance. This Guidance: (a) is for informational purposes only, (b) represents AWS current product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied. AWS responsibilities and liabilities to its customers are controlled by AWS agreements, and this Guidance is not part of, nor does it modify, any agreement between AWS and its customers.*
+

guidance/guidance-ec2.yml

@@ -1,6 +1,3 @@
-# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
-# SPDX-License-Identifier: MIT-0
-
 AWSTemplateFormatVersion: '2010-09-09'
 Transform: 'AWS::Serverless-2016-10-31'
 Description: Create and EC2 to demonstrate the benefits of pairing RDBMS workloads with AWS ElastiCache
@@ -74,6 +71,7 @@ Resources:
       ImageId: ami-0d8f91fa8ecdc3b58
       KeyName: !Ref AppInstanceKeyName
       Monitoring: true
+      DisableApiTermination: true
       SecurityGroupIds:
         - !Ref AppInstanceSecurityGroup
       SubnetId: !Ref AppInstanceSubnet
@@ -88,4 +86,4 @@ Resources:
       Tags:
         -
           Key: Name
-          Value: !Ref AppInstanceName
+          Value: !Ref AppInstanceName

guidance/requirements.txt

@@ -1,13 +1,111 @@
-python-decouple==3.8
-redis==5.0.4
-SQLAlchemy==2.0.30
-jupyter-events==0.6.3
-jupyter-server==1.24.0
+aiofiles==22.1.0
+aiosqlite==0.19.0
+anyio==3.7.1
+argon2-cffi==23.1.0
+argon2-cffi-bindings==21.2.0
+arrow==1.2.3
+async-timeout==4.0.3
+attrs==23.2.0
+Babel==2.14.0
+backcall==0.2.0
+beautifulsoup4==4.12.3
+bleach==6.0.0
+cached-property==1.5.2
+certifi==2024.2.2
+cffi==1.15.1
+charset-normalizer==3.3.2
+cycler==0.11.0
+debugpy==1.7.0
+decorator==5.1.1
+defusedxml==0.7.1
+entrypoints==0.4
+exceptiongroup==1.2.0
+fastjsonschema==2.19.1
+fonttools==4.38.0
+fqdn==1.5.1
+greenlet==3.0.3
+idna==3.6
+importlib-metadata==6.7.0
+importlib-resources==5.12.0
+ipykernel==6.16.2
+ipython==7.34.0
+ipython-genutils==0.2.0
+isoduration==20.11.0
+jedi==0.19.1
+Jinja2==3.1.3
+json5==0.9.16
+jsonpointer==2.4
+jsonschema==4.18.0
+jupyter-events==0.9.0
+jupyter-server==2.14.1
 jupyter-ydoc==0.2.5
 jupyter_client==7.4.9
 jupyter_core==4.12.0
 jupyter_server_fileid==0.9.1
 jupyter_server_ydoc==0.8.0
 jupyterlab==3.6.7
 jupyterlab-pygments==0.2.2
-jupyterlab_server==2.24.0
+jupyterlab_server==2.24.0
+kiwisolver==1.4.5
+markdown-it-py==2.2.0
+MarkupSafe==2.1.5
+matplotlib==3.5.3
+matplotlib-inline==0.1.6
+mdurl==0.1.2
+mistune==3.0.2
+nbclassic==1.0.0
+nbclient==0.7.4
+nbconvert==7.6.0
+nbformat==5.8.0
+nest-asyncio==1.6.0
+notebook==6.5.6
+notebook_shim==0.2.4
+numpy==1.21.6
+packaging==24.0
+pandas==1.3.5
+pandocfilters==1.5.1
+parso==0.8.3
+pexpect==4.9.0
+pickleshare==0.7.5
+Pillow==9.5.0
+pkgutil_resolve_name==1.3.10
+prometheus-client==0.17.1
+prompt-toolkit==3.0.43
+psutil==5.9.8
+ptyprocess==0.7.0
+pycparser==2.21
+Pygments==2.17.2
+PyMySQL==1.1.0
+pyparsing==3.1.2
+pyrsistent==0.19.3
+python-dateutil==2.9.0.post0
+python-decouple==3.8
+python-json-logger==2.0.7
+pytz==2024.1
+PyYAML==6.0.1
+pyzmq==24.0.1
+redis==5.0.3
+requests==2.31.0
+rfc3339-validator==0.1.4
+rfc3986-validator==0.1.1
+rich==13.7.1
+Send2Trash==1.8.2
+six==1.16.0
+sniffio==1.3.1
+soupsieve==2.4.1
+SQLAlchemy==2.0.29
+terminado==0.17.1
+tinycss2==1.2.1
+tomli==2.0.1
+tornado==6.2
+traitlets==5.9.0
+typing_extensions==4.7.1
+uri-template==1.3.0
+urllib3==1.26.6
+wcwidth==0.2.13
+webcolors==1.13
+webencodings==0.5.1
+websocket-client==1.7.0
+y-py==0.6.2
+ypy-websocket==0.8.4
+zipp==3.15.0

guidance/setup_jupyter.sh

@@ -1,19 +1,33 @@
 #!/bin/bash
 #
-# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
-# SPDX-License-Identifier: MIT-0
-
 # debug flags
-# set -ex
+#set -ex
 source .venv/bin/activate
 
-echo 'y' | jupyter notebook --generate-config
+# Save old configuration file if exits
+JUPYTER_CONFIG="$HOME/.jupyter/jupyter_lab_config.py"
+
+mv ${JUPYTER_CONFIG} ${JUPYTER_CONFIG}_$(date +%F-%T)
+
+echo 'y' | jupyter lab --generate-config 
+
+read -s -p "Password: " PASSWORD 
+
+echo ""
+echo "entered passwrord ${PASSWORD}"
+PASSWORD=$(python -c "from jupyter_server.auth import passwd; print(passwd('${PASSWORD}'))")
+
+echo "before"
+cat ${JUPYTER_CONFIG} | grep 'c.ServerApp.password'
+
+sed -i "s|# c.ServerApp.password = ''|c.ServerApp.password  = '${PASSWORD}'|g"   ${JUPYTER_CONFIG} 
 
-PASSWORD=$(python -c "from jupyter_server.auth import passwd; print(passwd('test123'))")
+echo "after"
+cat ${JUPYTER_CONFIG} | grep 'ServerApp.password'
 
-sed -i "s|# c.NotebookApp.password = ''|c.NotebookApp.password = '${PASSWORD}'|g"   ~/.jupyter/jupyter_notebook_config.py
-sed -i "s/# c.NotebookApp.ip = 'localhost'/c.NotebookApp.ip='0.0.0.0'/g"            ~/.jupyter/jupyter_notebook_config.py
-sed -i "s/# c.NotebookApp.port = 8888/c.NotebookApp.port = 8888/g"                  ~/.jupyter/jupyter_notebook_config.py
-sed -i "s/# c.NotebookApp.open_browser = True/c.NotebookApp.open_browser = False/g" ~/.jupyter/jupyter_notebook_config.py
+sed -i "s/# c.ServerApp.allow_password_change = True/c.ServerApp.allow_password_change = True/g" ${JUPYTER_CONFIG}
+sed -i "s/# c.ServerApp.ip = 'localhost'/c.ServerApp.ip='0.0.0.0'/g" 	    		        ${JUPYTER_CONFIG} 
+sed -i "s/# c.ServerApp.port = 8888/c.ServerApp.port = 8888/g"		    		        ${JUPYTER_CONFIG} 
+sed -i "s/# c.ServerApp.open_browser = True/c.ServerApp.open_browser = False/g"  	        ${JUPYTER_CONFIG} 
 
 jupyter lab &