Define specific IAM roles and policies

broker/iam/README.md

@@ -0,0 +1,211 @@
+# IAM Resources<a name="iam-resources"></a>
+
+<!-- mdformat-toc start --slug=github --maxlevel=6 --minlevel=1 -->
+
+- [IAM Resources](#iam-resources)
+  - [Setup](#setup)
+  - [Roles](#roles)
+    - [Create the roles](#create-the-roles)
+    - [Update a role](#update-a-role)
+  - [Policies](#policies)
+    - [Set the policy on the project](#set-the-policy-on-the-project)
+    - [Set a policy on a specific resource](#set-a-policy-on-a-specific-resource)
+  - [Onboard a new developer](#onboard-a-new-developer)
+    - [Developer instructions](#developer-instructions)
+    - [Project manager instructions](#project-manager-instructions)
+    - [Offboard a developer](#offboard-a-developer)
+
+<!-- mdformat-toc end -->
+
+Basic idea:
+
+We need to connect three things: resource(s), permission(s), and user(s).
+
+1. Create a Role - this is a collection of permissions and is a registered resource in GCP.
+1. Create a Policy - this is a local file that defines a collection of bindings (bindings attach roles to members)
+1. Attach the Policy to the Project, or to a specific Resource
+
+Helpful links:
+
+- [IAM overview](https://cloud.google.com/iam/docs/overview)
+- [IAM predefined roles reference](https://cloud.google.com/iam/docs/understanding-roles#predefined)
+- [IAM permissions reference](https://cloud.google.com/iam/docs/permissions-reference)
+
+## Setup<a name="setup"></a>
+
+Set GCP environment variables and authenticate yourself
+For reference, our current projects are:
+
+- production project: `ardent-cycling-243415`
+- testing project: `avid-heading-329016`
+
+```bash
+# Set environment variables
+# fill these in with your values; they are used throughout
+export GOOGLE_CLOUD_PROJECT=<project_id>
+export GOOGLE_APPLICATION_CREDENTIALS=<path/to/GCP_auth_key.json>
+
+# Authenticate to use gcloud tools in this project
+gcloud auth activate-service-account \
+    --project="${GOOGLE_CLOUD_PROJECT}" \
+    --key-file="${GOOGLE_APPLICATION_CREDENTIALS}"
+```
+
+## Roles<a name="roles"></a>
+
+### Create the roles<a name="create-the-roles"></a>
+
+Here is an example for creating the developer role.
+
+```bash
+role_id=developer
+role_yaml=roles/developer.yaml
+gcloud iam roles create $role_id --project=$GOOGLE_CLOUD_PROJECT --file=$role_yaml
+```
+
+### Update a role<a name="update-a-role"></a>
+
+(simpler in python so you don't have to mess with the etag fingerprint)
+
+```python
+# this is an untested example from the following url
+# https://cloud.google.com/iam/docs/creating-custom-roles
+def edit_role(name, project, title, description, permissions, stage):
+    """Creates a role."""
+
+    # pylint: disable=no-member
+    role = service.projects().roles().patch(
+        name='projects/' + project + '/roles/' + name,
+        body={
+            'title': title,
+            'description': description,
+            'includedPermissions': permissions,
+            'stage': stage
+        }).execute()
+
+    print('Updated role: ' + role['name'])
+    return role
+```
+
+## Policies<a name="policies"></a>
+
+We should set the project policy when the project is created.
+Afterwards, add or remove bindings to the policy individually (see onboarding section below for an example) rather than using the instructions in this section (which set the policy as a whole).
+
+### Set the policy on the project<a name="set-the-policy-on-the-project"></a>
+
+This binds member-role pairs to every resource in the project. (See below to bind to a specific resource.)
+
+We must download the current policy, update the file and use it to set a new policy.
+Setting a policy overrides the current policy.
+
+```bash
+policy_file="current_policy.yaml"
+gcloud projects get-iam-policy $GOOGLE_CLOUD_PROJECT >> $policy_yaml
+# update the bindings in the policy_file as needed, then set a new policy
+gcloud projects set-iam-policy $GOOGLE_CLOUD_PROJECT $policy_yaml
+```
+
+### Set a policy on a specific resource<a name="set-a-policy-on-a-specific-resource"></a>
+
+(simpler in python so you don't have to mess with the etag fingerprint)
+
+```python
+# this is an example from
+# https://cloud.google.com/pubsub/docs/access-control#python_2
+from google.cloud import pubsub_v1
+
+# TODO(developer): Choose an existing subscription.
+# project_id = "your-project-id"
+# subscription_id = "your-subscription-id"
+
+client = pubsub_v1.SubscriberClient()
+subscription_path = client.subscription_path(project_id, subscription_id)
+
+policy = client.get_iam_policy(request={"resource": subscription_path})
+
+# Add all users as viewers.
+policy.bindings.add(role="roles/pubsub.viewer", members=["domain:google.com"])
+
+# Add a group as an editor.
+policy.bindings.add(role="roles/editor", members=["group:[email protected]"])
+
+# Set the policy
+policy = client.set_iam_policy(
+    request={"resource": subscription_path, "policy": policy}
+)
+
+print("IAM policy for subscription {} set: {}".format(subscription_id, policy))
+
+client.close()
+```
+
+## Onboard a new developer<a name="onboard-a-new-developer"></a>
+
+NOTE: The following instructions are left for reference, but these are in the docs (currently at: docs/source/broker/initial-setup/manager-instructions.rst) and should be kept up-to-date there.
+
+### Developer instructions<a name="developer-instructions"></a>
+
+Complete the instructions in the initial setup tutorial
+(docs/source/broker/initial-setup/initial-setup.rst).
+
+You will need to do this in conjunction with a Pitt-Google project manager so that they can grant you the necessary permissions.
+
+You will need to provide the manager with both your Google account email address (e.g., Gmail address) and your service account name (which you will choose during setup).
+
+### Project manager instructions<a name="project-manager-instructions"></a>
+
+Make sure you've authenticated with the GCP project that the developer needs access to (see [Setup](#setup)).
+
+Note this needs to be done in conjunction with the developer's setup.
+The developer needs permissions (a role) bound to their Google account in order to create a service account, and the service account must exist before we can bind a role to it.
+
+Setup:
+
+```bash
+# fill in the user's Google account email address (e.g., Gmail address):
+user_email=
+
+# fill in the user's service account name and set the email address
+# service_account_name=
+service_account_email="${service_account_name}@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com"
+
+# choose a role_id (one example is commented out below), and set the role
+role_id=
+# role_id="developer"
+role="projects/${GOOGLE_CLOUD_PROJECT}/roles/${role_id}"
+# the above syntax is for a custom role that we've defined in the project
+# you can also use a predefined role. see the reference link given above for all options
+# role="role/viewer"
+```
+
+Add two policy bindings; one for the user's Google account (e.g., Gmail address, used for console access, etc.) and one for the user's service account (used to make API calls).
+
+```bash
+gcloud projects add-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \
+    --member="user:${user_email}" \
+    --role="${role}"
+
+# the service account needs to exist before
+# we can run this command to bind the policy
+gcloud projects add-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \
+    --member="serviceAccount:${service_account_email}" \
+    --role="${role}"
+
+# required to deploy Cloud Functions. only needs to be granted to the user account.
+gcloud iam service-accounts add-iam-policy-binding [email protected] --member=user:[email protected] --role=roles/iam.serviceAccountUser
+```
+
+### Offboard a developer<a name="offboard-a-developer"></a>
+
+Follow the setup in [Project manager instructions](#project-manager-instructions), then remove both policy bindings:
+
+```bash
+gcloud projects remove-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \
+    --member="user:${user_email}" \
+    --role="${role}"
+
+gcloud projects remove-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \
+    --member="serviceAccount:${service_account_email}" \
+    --role="${role}"
+```

broker/iam/policy_examples/README.md

@@ -0,0 +1,3 @@
+# Policies
+
+The files in this directory are example policies. They are not currently set on our projects.

broker/iam/policy_examples/production_project.yaml

@@ -0,0 +1,8 @@
+# etag must be obtained from the current policy and filled in below
+etag:
+bindings:
+- role: "projects/ardent-cycling-243415/roles/userPublic"
+  members:
+  # allUsers is anyone on the internet, no authentication required.
+  # Note that access to some GCP services does require authentication.
+    - user:allUsers

broker/iam/policy_examples/testing_project.yaml

@@ -0,0 +1,8 @@
+# etag must be obtained from the current policy and filled in below
+etag:
+bindings:
+- role: "projects/avid-heading-329016/roles/developerStudent"
+  members:
+  # add users or service accounts.
+  # syntax for service accounts:
+  # - serviceAccount:<sa_name>@<project_id>.iam.gserviceaccount.com

broker/iam/roles/developer.yaml

@@ -0,0 +1,10 @@
+title: "Pitt-Google Developer"
+# intended role-id is (set this at deployment):
+# "developer"
+description: "Role granting permissions for Pitt-Google developers that are not already bundled in a suitable, predefined role."
+stage: "ALPHA"
+includedPermissions:
+- storage.buckets.get
+- storage.buckets.list
+- storage.objects.list
+- storage.objects.get

broker/iam/roles/user_public.yaml

@@ -0,0 +1,7 @@
+title: "Public User of Pitt-Google Broker"
+# intended role-id is (set this at deployment):
+# "userPublic"
+description: "Role graning permissions necessary for accessing Pitt-Google's public data resources. Intended to be bound to the `allUsers` member in Pitt-Google's production project. (allUsers is anyone on the internet, no authentication required. Note that access to some GCP services does require authentication."
+stage: "ALPHA"
+includedPermissions:
+- pubsub.topics.attachSubscription

docs/source/broker/initial-setup.rst

@@ -0,0 +1,8 @@
+Initial Setup
+========================
+
+.. toctree::
+   :maxdepth: 1
+
+   initial-setup/initial-setup
+   initial-setup/manager-instructions

docs/source/broker/run-a-broker-instance/initial-setup.rst → docs/source/broker/initial-setup/initial-setup.rst

@@ -17,6 +17,10 @@ project.
 Create a GCP Project
 --------------------
 
+.. note::
+
+	You do not need to complete this section if you are a new developer for Pitt-Google broker. You will use our GCP projects; you do not need to create your own. Skip to the Setup Local Environment section.
+
 1. Create a new Google Cloud Platform (GCP) project.
 
 -  A. Go to the `Cloud Resource
@@ -44,8 +48,13 @@ Setup Local Environment
 
 Broker instances *run* 100% in the Google Cloud, as determined by the
 code in the ``broker`` package. You can *develop* the code and/or
-*deploy* an instance to the Cloud from your local machine. Setup your
-environment as follows:
+*deploy* an instance to the Cloud from your local machine.
+
+.. note::
+
+	If you are a new developer for Pitt-Google, You will need to complete this in conjunction with a Pitt-Google manager so they can give you appropriate permissions.
+
+Setup your environment as follows:
 
 1. **Install Google Cloud SDK command-line tools** using one of the
    following options. Included tools: gcloud, gsutil, and
@@ -61,10 +70,11 @@ may be asked to re-authenticate occasionally in the future.
 
 .. code:: bash
 
-    PROJECT_ID=my-pgb-project  # replace with your GCP Project ID
+    # fill in the ID of the GCP project you created above (or Pitt-Google's project ID)
+    PROJECT_ID=<project_id>
 
-    gcloud auth login  # follow the instructions to login to GCP
-    gcloud config set project $PROJECT_ID  # set your project ID
+    gcloud auth login  # follow the instructions to login to GCP with a Google account
+    gcloud config set project $PROJECT_ID  # set gcloud to use this project by default
 
 2. **Install Python libraries** for `GCP
    services <https://cloud.google.com/python/docs/reference>`__ and
@@ -80,8 +90,8 @@ may be asked to re-authenticate occasionally in the future.
     conda create -n pgb python=3.7
     conda activate pgb
 
-    # install the requirements. assumes txt file is in current directory
-    pip3 install -r requirements.txt
+    # install the pgb-broker-utils library, which also installs several google.cloud libraries
+    pip3 install pgb-broker-utils
 
 **Note**: On an M1 Mac, first use Conda to install Astropy
 (``conda install astropy=3.2.1``), then comment the related line out of
@@ -94,50 +104,58 @@ the requirements file before doing ``pip install``.
 
 .. code:: bash
 
-    PROJECT_ID=my-pgb-project  # replace with your GCP Project ID
-    NAME=my-service-account  # replace with desired account name
-    KEY_PATH=local/path/GCP_auth_key.json  # replace with desired path (ending in .json)
+    # choose a service account name (e.g., your name) and fill it in
+    SA_NAME=<service-account-name>
+    # choose a local path to store your authentication file and fill it in (file name must end with .json)
+    KEY_PATH=<local/path/GCP_auth_key.json>
 
-    gcloud iam service-accounts create $NAME
-    gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:$NAME@$PROJECT_ID.iam.gserviceaccount.com" --role="roles/owner"
-    gcloud iam service-accounts keys create $KEY_PATH --iam-account=$NAME@$PROJECT_ID.iam.gserviceaccount.com
+    # create the service account
+    gcloud iam service-accounts create $SA_NAME
 
-4. **Set environment variables**
+    # If this is a Pitt-Google project, send your service account name (SA_NAME)
+    # to a project manager to and as them to grant you a "developer" role on the project
+    # Otherwise, assign a role to your service account.
+    # This example below assigns a predifined role called "editor"
+    # gcloud projects add-iam-policy-binding "$GOOGLE_CLOUD_PROJECT" \
+    #     --member="serviceAccount:${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
+    #     --role="role/editor"
 
-.. code:: bash
+    # download the authentication file
+    gcloud iam service-accounts keys create $KEY_PATH --iam-account="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
 
-    PROJECT_ID=my-pgb-project  # replace with your GCP Project ID
-    KEY_PATH=local/path/GCP_auth_key.json  # same path as in step 3
+4. **Set environment variables**
 
+.. code:: bash
 
     export GOOGLE_CLOUD_PROJECT="$PROJECT_ID"
     export GOOGLE_APPLICATION_CREDENTIALS="$KEY_PATH"
     # export CLOUDSDK_COMPUTE_ZONE=
 
-If you are using a Conda environment, you can configure the environment
-variables as follows:
+If you are using a Conda environment, you can configure it to automatically set these environment
+variables when you activate the environment as follows:
 
 .. code:: bash
 
-    PROJECT_ID=my-pgb-project  # replace with your GCP Project ID
-    KEY_PATH=local/path/for/key/file.json  # same path as in step 3
-
-    # log into the environment and create de/activate files
+    # log into the environment and create activate and deactivate files
     conda activate pgb
     cd $CONDA_PREFIX
     mkdir -p ./etc/conda/activate.d
     mkdir -p ./etc/conda/deactivate.d
     touch ./etc/conda/activate.d/env_vars.sh
     touch ./etc/conda/deactivate.d/env_vars.sh
 
-    # add environment variables
+    # add commands to automatically set these variables when the environment is activated
     echo "export GOOGLE_CLOUD_PROJECT='$PROJECT_ID'" >> ./etc/conda/activate.d/env_vars.sh
     echo "export GOOGLE_APPLICATION_CREDENTIALS='$KEY_PATH'" >> ./etc/conda/activate.d/env_vars.sh
+
+    # add commands to automatically unset these variables when the environment is deactivated
     echo 'unset GOOGLE_CLOUD_PROJECT' >> ./etc/conda/deactivate.d/env_vars.sh
     echo 'unset GOOGLE_APPLICATION_CREDENTIALS' >> ./etc/conda/deactivate.d/env_vars.sh
 
 5. **Check that your authentication works** by making an API request.
-   Here we request a list of Cloud Storage buckets (in Python):
+   The example below requests a list of Cloud Storage buckets (in Python):
+
+(This will not work until your service account is assigned to a role, per instructions in step 3)
 
 .. code:: python