Skip to content
/ redisEnterpriseVault Public

Testing Redis Enterprise with HashiCorp Vault plugin Redis Connect and Postgres

7 stars 1 fork Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

jphaugla/redisEnterpriseVault

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

102 Commits
ansible-openshift
ansible-openshift
 
 
demo
demo
 
 
images
images
 
 
perftest
perftest
 
 
postgres
postgres
 
 
rdi
rdi
 
 
redis-connect
redis-connect
 
 
terraform
terraform
 
 
tools
tools
 
 
vault
vault
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 
redis-connect-k8s.excalidraw
redis-connect-k8s.excalidraw
 
 
setEnvironment.sh
setEnvironment.sh
 
 

Repository files navigation

redisEnterpriseVault - Redis Enterprise integrated with Hashicorp Vault to migrate data from postgresql using redis connect

Purpose

Redis Connect Demo integrating Redis Enterprise and Postgresql with Hashicorp Vault with all components running in kubernetes. Optional path is included to deploy without Vault.

 

Outline

 

Overview

Set up full set of tools to do redis connect or RDI between postgresql and redis enterprise using GKE cluster or Openshift cluster and vault. All software pieces will run in separate namespaces in a GKE or Openshift cluster. This github also supports setting up an active/active k8s cluster. Solution Diagram

Important Links

Technical Overview

  • Can use terraform/ansible automation to do complete automated install and configuration of this environment
  • Or, follow the instructions using link above to "Set up vault on GKE" or "Installing OpenShift on GCP".
    • At this point, these manual steps are stale as emphasis has been on the automated steps.
  • Install Redis Enterprise on k8s using "Redis Enterprise k8s" link
  • Set up Postgresql using Kubegres
  • If using Vault, Setup Vault and "Hashicorp Vault plugin on Redis Enterprise k8s"
  • Work through Redis Connect or RDI using "With" or "Without" vault. Note: Have not tried RDI with vault

 

 

Instructions

Choose to do GKE or do OpenShift as kickoff script is completely different *IMPORTANT NOTE: Creating this demo application in your GCP account will create and consume GCP resources, which will cost money.

GKE Automated Instructions

This terraform ansible setup has been tested on an AMD64 and ARM64 mac. It needs some additional pip installs

NOTE you must have posgtresql binaries installed before pip3 install psycopg2

  • apply additional pip installs
pip3 install requests
pip3 install google-auth
pip3 install kubernetes
pip3 install psycopg2
  • Troubleshooting on the psycopg2 install. this was not needed on arm64 , arm64 needed postresql 14 installed to prevent error on pip3 install psycopg2. Used brew install postgresql@14 on arm64. 
pip3 install --global-option=build_ext \
            --global-option="-I/usr/local/opt/openssl/include" \
            --global-option="-L/usr/local/opt/openssl/lib" psycopg2

Run GKE

  • set the parameters in the main terraform job file
    • do_vault, do-postgres, do_redis_connect, and do_rdi are main control switches for what pieces of the ansible run
    • main.tf file includes version information such as redis enterprise version, vault chart version
      • it is very important to get the desired redis-enterprise version or the initial install will fail
      • check the redis enterprise database version and ensure modules have the correct versions
        • click on the link above containing the v to get redis enterprise database version from the Images section
        • go to redis enterprise release notes for this database version
        • scroll down to the Redis Stack for this version and note the versions for timeseries module
        • update redis-meta.yml for the correct timeseries version found above
        • Gears is a separate download and installation into the redis nodes so need to adjust gears version in several places
      • to check the vault versions use this hashicorp link
    • no need to set the variables in main parameter file
    • this parameter file is only needed for reruns to disable parts of the operation

Terraform creation

kick off the terraform creation-the gke creation takes a very long time-over 10 minutes

cd terraform/test
terraform init
terraform apply --auto-approve

GKE console access

GKE console access is available through the GCP console kubernetes link. The cluster name comes from the cluster_name_final terraform variable which is held in

Terraform destroy

  • to remove everything
    • can have issues with the deletion of GKE. Run delete script first.
cd demo
./destroy.sh
cd ../terraform/test
terraform destroy --auto-approve

Terraform troubleshooting

  • many errors are just terraform timing issues so can just repeat the apply or destroy and it will work
  • output such as vault keys will be in ./terraform/ansible-gke/temp
  • Can control which terraform pieces are built using the variable in this file terraform/ansible-gke/gke-test/vars/main.yml
  • terraform destroy has issues. Manual deletion of gke cluster is often needed as destroy fails

General notes

There are several locations for parameters.

  • The first is in terraform/test/main.tf This is only for GKE
    • these four boolean parameters control whether vault, postgres, redis-connect, and redis data integration are created: do_vault, do_postgres, do_redis_connect, do_rdi
  • When running GKE or Openshift manually, these same parameters are important in ./terraform/ansible-gke
    • see the files manual_run_gke.sh or manual_run_openshift.sh for the environment settings in these scripts
    • NOTE: parameters don't have the do_ prefix so paraemeters are gke, vault, postgres, redis_connect, and rdi
  • The second is in terrafrom/ansible-gke/gke-test/vars/main.yml This is for both GKE and OpenShift
    • The parameters in vars/main.yml control which tasks run
    • By manipulating these parameters certain parts of the ansible can be run leaving other parts intact
    • Combine the use of parameters with the deletion of a namespace can selectively rebuild part a namespace
    • kubectl delete namespace vault and kubectl delete namespace redis-connect work very well
    • after deleting the namespace, set all the variables in vars/main.yml to False except for the task rebuilding the deleted namespace
  • terraform parameters are copied to the ansible environment in terraform/provisioning.tf
  • Even if no changes are needed to the GKE cluster, the ansible deployment is kicked off by the terraform script.

OpenShift Automated Instructions

OpenShift automation does not use terraform at all. Openshift automation only uses ansible because the actual installation of the OpenShift cluster is automated by Openshift (using terraform internally) by installing client software and using this client to automate the install on GCP. For simplicity, this GitHub automates this installation using ansible. Following the use of ansible to install OpenShift, the same ansible that is used in the GKE automation of the Redis Enterprise, PostgreSQL, Hashicorp Vault, and Redis Connect is also used with OpenShift. There are a few minor differences but parametrization takes care of the differences.

Run ansible OpenShift

cp ansible-openshift/gcp/vars/main.yml.template ansible-openshift/gcp/vars/main.yml

Edit main.yml to set your environment. There are many REPLACEME entries you must replace with your environment.

You'll also need to export the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to your service account key, e.g.,

export GOOGLE_APPLICATION_CREDENTIALS=/Users/<your home dir>/.gcp/central-beach-194106-abcdefg.json
cd ansible-openshift
./manual.sh

When the above script finishes it will output an export KUBECONFIG command. Use this command to allow the oc and/or kubectl commands to work. These binaries (oc and kubectl) are in the ./ansible-openshift/binaries directory

  • Can find all of these values in ansible-openshift/install-files/.openshift_install.log. Can find the full openshift console address, password for kubeadmin, and KUBECONFIG environment variable.
  • Openshift console access is available using https://console-openshift-console.apps.<ocp-cluster_name>.<openshift_cluster_base_domain>
  • Username is kubeadmin and the password is in the file

Run ansible k8s

Run ansible k8s steps to configure postgres, redis enterprise, etc

  • Verify the parameters in main parameter file. These should only need to be changed on rerun/error conditions
  • Check the ansible script environment variables
  • it is very important to know the desired redis-enterprise version to have the correct module versions as RDI needs redis gears and redis connect needs timeseries
  • This openshift install doesn't really allow the choice of the redis operator and redis database versions. Install relies on cloning the redis-enterprise k8s repository and using the version in this repository. If you want an older version, need to clone that earlier version.
  • To see the redis operator version in the github, check this file openshift.bundle.yaml and search for this string in the file redis-enterprise-operator:
  • check the redis enterprise database version and ensure modules have the correct versions *Click on the button releases
    • enter the release with the tag (it starts with a v)
    • click on the link above containing the v to get redis enterprise database version from the Images section
    • go to redis enterprise release notes for this database version
    • scroll down to the Redis Stack for this version and note the versions for timeseries module
    • update redis-meta.yml for the correct timeseries version found above
    • Gears is a separate download and installation into the redis nodes so need to adjust gears version in several places
  • to check the vault versions use this hashicorp link
    • verify you have the vault, postgres, redis_connect and rdi variable set appropriately
    • verify the redis_enterprise version
  • Kick off the ansible script
# better to use the full path here from KUBECONFIG returned above
export KUBECONFIG=./ansible-openshift/install-files/auth/kubeconfig
cd  terraform/ansible-gke
./manual_run_openshift.sh

At the end of the ansible execution, the current files are copied over to ./terraform/ansible-gke/temprun-<rec_name>

Destroy openshift environment

cd ./ansible-openshift
./binaries/openshift-install destroy cluster --dir install-files

Active-Active

Many of the pieces are in place for active/active in this github. The steps are taken from this redis k8s active/active link

  • the prepare clusters steps are completed
    • Admission controller is configured
    • If openshift, route is created. If GKE, haproxy is created
    • Necessary DNS entry is created in Cloud DNS to access the cluster from the route or haproxy
    • The new secret for the cluster is created as redis-enterprise-rerc-test-rec-1 where test-rec-1 is the redis enterprise rec name
      • ./terrafrom/ansible-gke/temprun-<rec_name>/demo/rerc-secrets-<rec_name>.yaml contains the secret contents. This has been applied to the current cluster but will be needed for the other cluster(s)
    • The rerc record yaml is also created for this cluster but has not been applied because the rerc files only need to be applied to one of the participating clusters
      • ./terrafrom/ansible-gke/temprun-<rec_name>/demo/rerc-<rec_name>.yaml contains the rerc definition
    • To switch between environments using openshift is just setting the KUBECONFIG environment variable
      • (first cluster) export KUBECONFIG=/Users/jason.haugland/gits/redisEnterpriseVault/ansible-openshift/install-files/auth/kubeconfig
      • (second cluster) export KUBECONFIG=/Users/jason.haugland/gits/redisEnterpriseVault/ansible-openshift/install-files-2/auth/kubeconfig
    • Verify can connect to both clusters
      • edit demo/testroute one cluster
      • test that cluster
cd demo
./testroute.sh
* repeat for second cluster

Set environment to first cluster

export KUBECONFIG=/Users/jason.haugland/gits/redisEnterpriseVault/ansible-openshift/install-files/auth/kubeconfig

Create remote cluster secret

kubectl -n demo apply -f temprun-test-rec-2/demo/test-rec-2-rerc-secrets.yaml

Create remote clusters

cd terraform/ansible-gke/
kubectl -n demo create -f temprun-test-rec-1/demo/rerc-test-rec-1.yaml
kubectl -n demo create -f temprun-test-rec-2/demo/rerc-test-rec-2.yaml
# verify rerc is good
kubectl -n demo get rerc

Output should match

Run manually

These manual steps have not been actively maintained and are stale at this point.

 

There are a large number of directories to keep track of files. A provided script will set up the following environment variables to make the task less arduous.
Another tip would be to have a window opened to each of the directories instead of relying on one terminal session.
Additionally, label each terminal session for the directory path in use.

Prepare repository working directories

To get all of these moving parts working, multiple repositories are needed. Then, a large number of directory changes are needed as different pieces are deployed.
To facilitate this, first set up the environment with a provided scripts and then pull all the necessary repositories. Decide on one home git directory that will hold all the subdirectories needed. The default in the environment script is $HOME/gits

  • Move to chosen git directory home and pull down the repositories
git clone https://github.com/jphaugla/redisEnterpriseVault.git
git clone https://github.com/RedisLabs/redis-enterprise-k8s-docs.git
git clone https://github.com/redis-field-engineering/redis-connect-dist.git
  • edit and then source the environment files for subsequent steps
cd redisEnterpriseVault
source setEnvironment.sh

Choose GKE or OpenShift

Create GKE cluster

Tips on installing GKE

  • Easier to use GCP console to get the desired node size.
  • Make sure compute nodes are decent size e2-standard-8.
  • Start with 3 nodes in the default node pool-can always increase as needed.
  • Once the GKE cluster is created, connect to the cluster. To do this:
    • Click on newly created cluster Select GKE Cluster
    • Click to connect to the cluster Connect to Cluster
    • Follow the command-line access instructions to prepare for the subsequent steps

Create OpenShift cluster

Install Redis Enterprise k8s

  • Get to redis enterprise k8s docs directory
cd $GIT_RE_K8S

Create redis enterprise databases

  • A separate role is created for each database. This is VERY IMPORTANT for effectively using the plugin. Each database needs a separate role only for that database. If a common role is used across all databases in a large cluster, the performance aspect can be very significant,
  • Create two redis enterprise databases. The first database is the Target database for redis connect and the second stores meta-data for redis-connect
    • If, the redis-meta database doesn't create, it may be the version of the timeseries module as it must fit with the deployed version. Verify the module version for this redis enterprise version using the release notes
* get the cluster password for use in the curl command substituting out the 5IcX7yYD with the correct password to create a role 
* the ports are hardcoded in the yaml files for each database, redis-enterprise-database is 12000 and redis-meta is 120001
```bash
./getClusterUnPw.sh

cd $DEMO
curl -v -k -u [email protected]:5IcX7yYD -X POST https://localhost:9443/v1/roles -H Content-type:application/json  -d '{"name":"db1","management":"admin"}'
curl -v -k -u [email protected]:5IcX7yYD -X POST https://localhost:9443/v1/roles -H Content-type:application/json  -d '{"name":"db2","management":"admin"}'
kubectl apply -f redis-enterprise-database.yml
kubectl apply -f redis-meta.yml
  • get the database password
./getDatabasePw.sh

Add Redisinsights

These instructions are based on Install RedisInsights on k8s   The above instructions have two options for installing redisinights, this uses the second option to install without a service (avoids creating a load balancer)

  • The yaml file apply below, creates redisinsights
  • Since no load balancer is deployed, must do a port-forward to be able to access redisinsights from the local machine's browser
  • Easiest, to open a new terminal window and label the terminal window redisinsights port forward
kubectl apply -f redisinsight.yml
kubectl port-forward deployment/redisinsight 8001

Install Kubegres

Based on the instructions so also read these as steps are performed for deeper explanation Kubegres getting started This creates kubegres, creates a postgres.conf configmap to enable postgres replication, adds postgres database and password, and creates the one node database The replication technique with the configmap uses this link Override default configs.

cd $POSTGRES
kubectl apply -f https://raw.githubusercontent.com/reactive-tech/kubegres/v1.16/kubegres.yaml
kubectl create namespace postgres
kubectl config set-context --current --namespace=postgres
kubectl apply -f postgres-conf-override.yaml
kubectl apply -f my-postgres-secret.yaml
kubectl apply -f my-postgres.yaml
  • create database and tables needed for redis-connect
    • find the pod name for postgres
    • copy database and table creation to pod
    • use the password in the my-postgres-secret.yaml file when prompted with psql
cd $SAMPLES
kubectl get pods
kubectl -n postgres cp postgres_cdc.sql mypostgres-1-0:/
kubectl -n postgres exec --stdin --tty  mypostgres-1-0 -- /bin/sh
psql -Upostgres -W
\i postgres_cdc.sql

Vault

Install helm and vault on GKE

cd $VAULT
kubectl create namespace vault
kubectl config set-context --current --namespace=vault
brew install helm
brew install jq
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
# this one is for gke
helm install vault hashicorp/vault --namespace vault -f override-values.yaml
# this one is for openshift
helm install vault hashicorp/vault --namespace vault -f override-values-openshift.yaml

unseal the vault

  • This follows the technique from within this link Set up vault on GKE. Read this section for explanation of these commands.
  • The operator init writes pertinent keys to the cluster-keys.json file for safe keeping, obtain unseal key, unseal the vault and display status
kubectl exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > cluster-keys.json
export VAULT_UNSEAL_KEY=$(cat cluster-keys.json | jq -r ".unseal_keys_b64[]")
kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY
kubectl exec vault-0 -- vault status
  • To use the vault, a token is needed. For this exercise, use the root token to login
cat cluster-keys.json | jq -r ".root_token"
export CLUSTER_ROOT_TOKEN=$(cat cluster-keys.json | jq -r ".root_token")
kubectl exec vault-0 -- vault login $CLUSTER_ROOT_TOKEN

Download the plugin file

Make sure you grab the correct file-many similarly named files

mv ~/Downloads/vault-plugin-database-redis-enterprise_0.1.3_linux_amd64 $VAULT
kubectl cp -n vault $VAULT/vault-plugin-database-redis-enterprise_0.1.3_linux_amd64 vault-0:/usr/local/libexec/vault
shasum -a 256 $VAULT/vault-plugin-database-redis-enterprise_0.1.3_linux_amd64| awk '{print $1}'
  • get the cluster and database password information for use while logged into vault
$DEMO/getDatabasePw.sh
$DEMO/getClusterUnPw.sh

log to vault container and enable vault plugin

Use the shasum value pulled from above and not the current value set equal to sha256

kubectl -n vault exec --stdin=true --tty=true vault-0 -- /bin/sh
vault write sys/plugins/catalog/database/redisenterprise-database-plugin command=vault-plugin-database-redis-enterprise_0.1.3_linux_amd64 sha256=739421599adfe3cdc53c8d6431a3066bfc0062121ba8c9c68e49119ab62a5759

Create database configurations in vault

Using the information from the getClusterUnPw.sh script from above and using the username and password valued for redis enterprise cluster authentication.
For additional explanations peruse Hashicorp Vault plugin on Redis Enterprise k8s

chmod 755 /usr/local/libexec/vault/vault-plugin-database-redis-enterprise_0.1.3_linux_amd64
vault secrets enable database
vault write database/config/demo-test-rec-redis-enterprise-database plugin_name="redisenterprise-database-plugin" url="https://test-rec.demo.svc:9443" allowed_roles="*" database=redis-enterprise-database [email protected] password=vubYurxK
vault write database/config/demo-test-rec-redis-meta plugin_name="redisenterprise-database-plugin" url="https://test-rec.demo.svc:9443" allowed_roles="*" database=redis-meta [email protected] password=vubYurxK

Create database roles

  • as mentioned above, is important each database uses its own role here for creating the dynamic usernames. Sharing one role in the cluster across databases can cause performance problems.
vault write database/roles/redis-enterprise-database db_name=demo-test-rec-redis-enterprise-database creation_statements="{\"role\":\"db1\"}" default_ttl=90m max_ttl=100m
vault write database/roles/redis-meta db_name=demo-test-rec-redis-meta creation_statements="{\"role\":\"db2\"}" default_ttl=90m max_ttl=100m

Authorize kubernetes

  • using vault terminal connection...
vault auth enable kubernetes
vault secrets enable kubernetes
vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  • don't log out of vault, keep vault connection in a separate terminal and use the vault terminal as directed

Redis Connect Configuration

  • create service account and namespace
    • service account is only needed if using vault integration
cd $REDIS_CONNECT
kubectl create namespace redis-connect
kubectl config set-context --current --namespace=redis-connect
kubectl create sa redis-connect

Redis Connect With Vault

not needed if not doing vault (skip to Redis Connect Without Vault)

  • go to vault terminal
vault write database/config/kube-postgres \
    plugin_name=postgresql-database-plugin \
    allowed_roles="redis-connect" \
    username="postgres" \
    password="jasonrocks" \
    connection_url="postgresql://{{username}}:{{password}}@mypostgres.postgres.svc:5432/RedisConnect?sslmode=disable"
vault write database/roles/redis-connect \
    db_name=kube-postgres \
    creation_statements="CREATE ROLE \"{{name}}\" WITH REPLICATION LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
         GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\"; \
         ALTER USER \"{{name}}\" WITH SUPERUSER;" \
    default_ttl="60m" \
    max_ttl="60m"
  • create redis-connect policy and role
vault policy write redis-connect-policy - <<EOF
 path "database/creds/redis-enterprise-database" {
   capabilities = ["read"]
 }
 path "database/creds/redis-meta" {
   capabilities = ["read"]
 }
path "database/creds/redis-connect" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/redis-connect \
    bound_service_account_names=redis-connect \
    bound_service_account_namespaces=redis-connect \
    policies=redis-connect-policy \
    ttl=24h
  • create configmap with jobmanager.properties
    • this configmap is used in the redis-connect-start.yaml to mount files appropriately
  • start the redis-connect server
$DEMO/getDatabasePw.sh
vi jobmanager.properties
kubectl create configmap redis-connect-config \
  --from-file=jobmanager.properties=jobmanager.properties 
  --from-file=logback.xml=logback.xml 
kubectl apply -f vault/redis-connect-start.yaml

Redis Connect Without Vault

  • Edit the jobmanager.properties file for the correct connection parameters in redis.connection.url
    • these parameters can be retrieved using $DEMO/getDatabasePw.sh
  • Edit each credentials file for the appropriate connection information
    • For redisconnect_credentials_jobmanager.properties (redis-meta) and redisconnect_credentials_redis_postgres-job.properties (redis-enterprise-db) can use a null username and the password from getDatabasePw.sh
  • create configmap with jobmanager.properties and the credentials files
    • this configmap is used in the redis-connect-start.yaml to mount files appropriately
  • start the redis-connect server
$DEMO/getDatabasePw.sh
vi jobmanager.properties
kubectl create configmap redis-connect-config \
  --from-file=jobmanager.properties=jobmanager.properties \
  --from-file=logback.xml=logback.xml 
  --from-file=redisconnect_credentials_jobmanager.properties=redisconnect_credentials_jobmanager.properties \
  --from-file=redisconnect_credentials_redis_postgres-job.properties=redisconnect_credentials_redis_postgres-job.properties \
  --from-file=redisconnect_credentials_postgresql_postgres-job.properties=redisconnect_credentials_postgresql_postgres-job.properties 
kubectl apply -f non-vault/redis-connect-start.yaml

Validate Environment

Validate database admin access

  • port forward from kubernetes service to local port
kubectl port-forward -n demo service/redis-enterprise-database 12000
kubectl port-forward -n demo service/redis-meta 12001
  • access the redis-enterprise-database (port 12000) and redis-meta (port 12001) using the passwords from demo/getDatabasepw.sh
demo/getDatabasepw.sh
redis-cli -p 12000 -a 0BPpoREJ
>set jason cool
redis-cli -p 12001 -a 0XyzJPH1
>set jason uncool

Validate redisinsights

  • port forward to access redisinsights and each database to connect to
kubectl port-forward -n demo deployment/redisinsight 8001
  • from chrome or firefox, open the browser using http://localhost:8001
  • Click "I already have a database"
  • Click "Connect to Redis Database"
  • Create Connection to target redis database with following parameter entries
Key Value
host redis-enterprise-database.demo
port 12000 (can get from demo/getDatabasepw.sh but is hardcoded)
name TargetDB
Username (leave blank)
Password DrCh7J31 (from demo/getDatabasepw.sh above)
  • click ok *repeat steps above for metadata database using following parameters
Key Value
host redis-meta.demo
port 12001 (can get from demo/getDatabasepw.sh but is hardcoded)
name metaDB
Username (leave blank)
Password FW2mFXEH (from demo/getDatabasepw.sh above)

Validate vault access

Only execute these steps if running with vault

  • port forward to access the databases from local machine port
  • port forward from kubernetes service to local port
kubectl port-forward -n demo service/redis-enterprise-database 12000
kubectl port-forward -n demo service/redis-meta 12001

NOTES:

  • the redis-cli command is using the username and password from the output of the vault read database command
  • the vault read command must be done from the vault terminal
    • can log in to the vault container using kubectl exec -n vault --stdin=true --tty=true vault-0 -- /bin/sh
  • From vault for redis-enterprise-database
vault read database/creds/redis-enterprise-database
      Key                Value
      ---                -----
      lease_id           database/creds/redis-enterprise-database/JVsEvOrtZfK46dMO7GWRxjTW
      lease_duration     90m
      lease_renewable    true
      password           blZxlE10AS-zy-UBbjdh
      username           v_root_redis-enterprise-database_sruv9v0fewy2rv4m1oxq_1646252982
  • From vault for redis-meta
vault read database/creds/redis-meta
      Key                Value
      ---                -----
      lease_id           database/creds/redis-meta/iahGKJ9XNkisGsyLgW89Ohpt
      lease_duration     90m
      lease_renewable    true
      password           Vfo2ajqBvWAVKFyI-ojR
      username           v_root_redis-meta_n2dkafecjttws9mzj9eg_1646411445

Open a new terminal window and test each database using the username and password values from the vault read database output

  • redis-enterprise-database
redis-cli -p 12000
>AUTH v_root_redis-enterprise-database_sruv9v0fewy2rv4m1oxq_1646252982 blZxlE10AS-zy-UBbjdh
  • redis-meta
redis-cli -p 12001
>AUTH v_root_redis-meta_n2dkafecjttws9mzj9eg_1646411445 Vfo2ajqBvWAVKFyI-ojR

Test redis connect replication

This section will define the redis-connect job using an API approach. For more detail on the redis-connect swagger interface, see this demo section in redis connection github In another terminal, need to port-forward the rest API interface to set up the actual job

  • first get the redis-connect pod name with kubectl get all
  • use the pod name to port-forward the port
kubectl -n redis-connect get all
kubectl -n redis-connect port-forward pod/redis-connect-59475bcdd4-nwv5v 8282

Use the swagger interface to define the job and control the job execution

  • Save the job configuration using swagger interface swagger
    • click on QUICK START
    • click on POST SAVE Job Configuration to bring up the API interface API
    • click on Try It Out
    • Enter the jobname of postgres-job
    • Click on Choose File and select redis-connect/postgres-job.json
    • Click on Execute
  • Insert rows using postgres container
kubectl -n postgres exec --stdin --tty  mypostgres-1-0 -- /bin/sh
psql -Upostgres -W
\c "RedisConnect"
INSERT INTO public.emp (empno, fname, lname, job, mgr, hiredate, sal, comm, dept) VALUES (1, 'Allen', 'Terleto', 'FieldCTO', 1, '2018-08-06', 20000, 10, 1);
INSERT INTO public.emp (empno, fname, lname, job, mgr, hiredate, sal, comm, dept) VALUES (2, 'Brad', 'Barnes', 'SA Manager', 1, '2019-08-06', 20000, 10, 1);
INSERT INTO public.emp (empno, fname, lname, job, mgr, hiredate, sal, comm, dept) VALUES (3, 'Virag', 'Tripathi', 'Troublemaker', 1, '2020-08-06', 20000, 10, 1);
INSERT INTO public.emp (empno, fname, lname, job, mgr, hiredate, sal, comm, dept) VALUES (4, 'Jason', 'Haugland', 'SA', 1, '2021-08-06', 20000, 10, 1);
INSERT INTO public.emp (empno, fname, lname, job, mgr, hiredate, sal, comm, dept) VALUES (5, 'Ryan', 'Bee', 'Field Sales', 1, '2005-08-06', 20000, 10, 1);
  • Run the initial bulk load of the job swagger start
    • Click on POST START Job to biring up the API interface API start
    • click on Try It Out
    • Enter the jobname of postgres-job
    • Enter the jobtype of load
  • Test the results
  • Run the stream load swagger start
    • Click on POST START Job to bring up the API interface API start
INSERT INTO public.emp (empno, fname, lname, job, mgr, hiredate, sal, comm, dept) VALUES (111, 'Simon', 'Prickett', 'Tech Advocate', 1, '2016-08-06', 20000, 10, 1);
INSERT INTO public.emp (empno, fname, lname, job, mgr, hiredate, sal, comm, dept) VALUES (112, 'Doug', 'Snyder','Territory Manager', 1, '2021-08-06', 20000, 10, 1);
INSERT INTO public.emp (empno, fname, lname, job, mgr, hiredate, sal, comm, dept) VALUES (113, 'Jason', 'Plotch', 'Territory Manager', 1, '2021-08-06', 20000, 10, 1);
INSERT INTO public.emp (empno, fname, lname, job, mgr, hiredate, sal, comm, dept) VALUES (114, 'Nail', 'Sirazitdinov', 'TAM', 1, '2010-08-06', 20000, 10, 1);
INSERT INTO public.emp (empno, fname, lname, job, mgr, hiredate, sal, comm, dept) VALUES (115, 'Manish', 'Arora', 'SA', 1, '2021-08-06', 20000, 10, 1);
  • Test the results

Debug Ideas

  • Redis-connect job documentation link
  • look for resulting rows in redis enterprise using redisinsight (see directions above)
  • There are multiple methods to debug the running job. Here are a few:
    • Find the pod name(s) for redis connect
    • get the logs for init and main container
kubectl -n redis-connect get pods
kubectl -n redis-connect logs pod/redis-connect-5cd54fcddd-8s5pj -c vault-agent-init
kubectl -n redis-connect logs pod/redis-connect-5cd54fcddd-8s5pj -c vault-agent
kubectl -n redis-connect logs pod/redis-connect-5cd54fcddd-8s5pj -c redis-connect
kubectl -n redis-connect exec --stdin --tty  pod/redis-connect-5cd54fcddd-8s5pj -- /bin/sh
  • log in to the pod and look at log files
  • there is a debug line in redis-connect-start.yaml that keeps the pod running even if there are connection errors-this is great for debug
  • NOTE: check the swagger UI for API commands that are easier to use than redisconnect.sh
  • Check the file systems mounted correctly
cd logs
# investigate the log files 
vi *
# login to cli
cd ../bin
./redisconnect.sh cli
> validate connection -t JDBCConnectionProvider -i RDBConnection
exit
df -h

this should be output from the df -h df ouput

  • Ensure each credential file is named correctly
  • Test the username/password from the credential file to ensure the connection works from the respective database

Troubleshooting k8s DNS

The install process automatically creates a dns utils pod to debug any networking issues To debug use these notes:

About

Testing Redis Enterprise with HashiCorp Vault plugin Redis Connect and Postgres

Topics

kubernetes redis openshift vault terraform postgresql gke-cluster redis-enterprise redis-connect kubegres

Resources

Readme
Activity

Stars

7 stars

Watchers

4 watching

Forks

1 fork
Report repository

Releases

No releases published

Packages

No packages published

Contributors 3

  •  
  •  
  •  

Languages