Skip to content

Latest commit

 

History

History
104 lines (79 loc) · 6.43 KB

ReadMe.md

File metadata and controls

104 lines (79 loc) · 6.43 KB
Raw

This repository creates a Docker container which runs the 18f analytics-reporter which powers analytics.usa.gov. The repository also contains a manifest.yml file which is used to bind a user provided service for environment variables to the container and to set config options. In order for the docker container to run on cloud.gov no-route: true and health-check-type: process must be included in the manifest file. Including the docker image and username are optional but allow for shortened cf push command.

The base image is the node alpine image. It copies and runs testscript.sh. The script installs the analytics-reporter found in /analytics-reporter. To pull the data variables ANALYTICS_REPORT_IDS, ANALYTICS_REPORT_EMAIL, and ANALYTICS_KEY needed. To write to an S3 bucket the AWS_REGION,AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_BUCKET and AWS_BUCKET_PATH environment variables are needed. These are stored in a user provided service instance {s3user-provided-service} and are accessible through VCAP_SERVICES. The script requires bash and jq to parse the VCAP_SERVICES and export the variables.

The analytics reporter is called every 900 seconds (equal to 15 minutes). It outputs 33 json files to https://s3-{AWS_REGION}.amazonaws.com/{AWS_BUCKET}/{AWS_BUCKET_PATH}/{json-file}.

User Provided Services (UPS):

Cloud front documentation states:

“Note: Do not use user-provided environment variables for security sensitive information such as credentials as they might unintentionally show up in cf CLI output and Cloud Controller logs. Use user-provided service instances instead. The system-provided environment variable VCAP_SERVICES is properly redacted for user roles such as Space Supporter and in Cloud Controller log files.”

Eight environment variables (ANALYTICS_REPORT_IDS, ANALYTICS_REPORT_EMAIL, ANALYTICS_KEY, AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_BUCKET, AWS_BUCKET_PATH) must be saved and provided as a UPS. The below instructions were found here: IBM Link for CUPS. To bind the UPS to the app services: - name: {s3user-provided-service} is included in the manifest.yml file.

First Time:

  • cf cups cupsTest -p {vcap_keys}.json
  • cf bind-service {cf-app-name} cupsTest
  • cf restage {cf-app-name}

Subsequent/Updates:

  • cf uups cupsTest -p {vcap_keys}.json
  • docker buildx use default
  • docker buildx build -t {docker-repo}/{image} --platform linux/amd64 .
  • docker push {docker-repo}/{image}
  • CF_DOCKER_PASSWORD={docker-password} cf push {cf-app-name}

S3 on Cloud.Gov

The output of the analytics reporter is stored on a S3 bucket in cloud.gov. The AWS_REGION,AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_BUCKET and AWS_BUCKET_PATH are also stored in VCAP_SERVICES. The analytics reporter already comes with a pre-configuered lightweight S3 publishing tool which just requires the above variables. The json files are stored at https://s3-{AWS_REGION}.amazonaws.com/{AWS_BUCKET}/{AWS_BUCKET_PATH}/{json-file} where the {json-file} is one of the 23 following options:
all-domains-30-days screen-size top-external-links-7-days
all-pages-realtime today top-external-links-yesterday
browsers top-cities-90-days top-landing-pages-30-days
device_model top-cities-realtime top-pages-30-days
devices top-countries-90-days top-pages-7-days
ie top-countries-realtime top-pages-realtime
language top-domains-30-days top-traffic-sources-30-days
last-48-hours top-domains-7-days traffic-sources-30-days
os-browsers top-downloads-yesterday users
os top-exit-pages-30-days windows-ie
realtime top-external-links-30-days windows

Variables, Keys & Credentials you will need

Docker Cloud.gov Google Analytics AWS
docker-repo cf-app-name ANALYTICS_REPORT_EMAIL AWS_ACCESS_KEY_ID
image cf-user ANALYTICS_KEY AWS_SECRET_ACCESS_KEY
json-file s3user-provided-service ANALYTICS_REPORT_IDS AWS_BUCKET
--- --- --- AWS_BUCKET_PATH
--- --- --- AWS_REGION

Create docker container for cloud.gov to write to S3

  1. cd /analytics-reporter-docker-container
  2. docker buildx use default
  3. docker buildx build -t {docker-repo}/{image} --platform linux/amd64 .
  4. docker push {docker-repo}/{image}
    • suggested to make repo private on hub.docker
  5. Push to cloud.gov:
    • from public repo:
      • cf push {image} --docker-image {docker-repo}/{image}
    • from private repo:
      • CF_DOCKER_PASSWORD={docker-password} cf push {image} --docker-image {docker-repo}/{image} --docker-username {docker-repo}
    • if using manifest.yml:
      • CF_DOCKER_PASSWORD={docker-password} cf push {cf-app-name}
  6. Make S3 bucket
    • cf create-service s3 basic-public-sandbox {s3user-provided-service}
    • cf bind-service {cf-app-name} {s3user-provided-service}
    • cf restage {cf-app-name}
    • cf create-service-key {s3user-provided-service} {cf-user}
    • cf service-key {s3user-provided-service} {cf-user}
  7. Create user provided service
    • create & save environment variables (Google Analytics and AWS Keys) in {vcap_keys}.json
    • cf cups analyticsReporterServices -p [path to : {vcap_keys}.json]
    • cf bind-service {cf-app-name} analyticsReporterServices
    • cf restage {cf-app-name}
  8. Edit manifest.yml
    • The manifest.yml files now needs to point to the s3 bucket you just created. Add:
      • services: -name: analyticsReporterServices
  9. docker buildx build -t {docker-repo}/{image} --platform linux/amd64 .
  10. docker push {docker-repo}/{image}
  11. Push to cloud.gov:
    • from public repo:
      • cf push {image} --docker-image {docker-repo}/{image}
    • from private repo:
      • CF_DOCKER_PASSWORD={docker-password} cf push {image} --docker-image {docker-repo}/{image} --docker-username {docker-repo}
    • if using manifest.yml:
      • CF_DOCKER_PASSWORD={docker-password} cf push {cf-app-name}
  12. Access via: