Merge pull request #228 from seanmil/add_catalog_v4

Add Puppetserver catalog v4 API support

Author: ahayworth

README.md

@@ -23,7 +23,7 @@ There are some [limitations](doc/limitations.md) to a catalog-based approach, me
 `octocatalog-diff` is currently able to get catalogs by the following methods:
 - Compile catalog via the command line with a Puppet agent on your machine (as GitHub uses the tool internally)
 - Obtain catalog over the network from PuppetDB
-- Obtain catalog over the network using the API to query a Puppet Master / PuppetServer (Puppet 3.x and 4.x supported)
+- Obtain catalog over the network using the API to query a Puppet Master / PuppetServer (Puppet 3.x through 6.x supported)
 - Read catalog from a JSON file
 
 ## Example

doc/advanced-puppet-master.md

@@ -8,7 +8,11 @@ Please note the following caveats:
 
 0. You will need to deploy your Puppet code to an environment on your Puppet Master prior to running `octocatalog-diff` for that environment. `octocatalog-diff` does not deploy code for you.
 
-0. You will need to configure authorization for one or more whitelisted certificates on your Puppet Master. The default permissions allow a node to retrieve its own catalog via the API, but you need a certificate for `octocatalog-diff` that permits it to retrieve any catalog. See the [Certificate authorization](#certificate-authorization) section below.
+0. You will need to configure authorization for one or more whitelisted certificates on your Puppet Master. The default permissions allow a node to retrieve its own catalog via the API, but you need a certificate for `octocatalog-diff` that permits it to retrieve any catalog. See the [Certificate authorization](#certificate-authorization) section below. If you are using Puppet Enterprise and use
+the Puppet Master v4 API you may also use a Puppet Enterprise RBAC token. The user owning the token will need the "Puppet Server Compile catalogs for remote nodes" permission.
+See the [PE RBAC Token Authorization](#pe-rbac-token-authorization) section below.
+
+0. If you are using the v2 or v3 PuppetServer APIs with Octocatalog-Diff to compile catalogs, then those catalogs and facts will be automatically stored in PuppetDB. However, when using the v4 PuppetServer API with Octocatalog-Diff, facts and catalogs are *not* automatically stored in PuppetDB - persistence is optional and may be enabled with the appropriate Octocatalog-Diff CLI flag. If your environment depends on the accuracy of exported resources or facts in PuppetDB, you may wish to upgrade and use the V4 API, to avoid unintentional side-effects.
 
 ## Command line options
 
@@ -18,11 +22,15 @@ The following command line options are used to retrieve a catalog from a Puppet
 | ------ | ----------- |
 | `-f ENVIRONMENT` | Environment name to use for the "from" catalog |
 | `-t ENVIRONMENT` | Environment name to use for the "to" catalog |
-| `--puppet-master HOSTNAME:PORT | The hostname and port number of the Puppet Master. (By default the port used by Puppet Master is 8140.) |
-| `--puppet-master-api-version VERSION | The API version used by the Puppet Master. API versions 2 and 3 are supported. Puppet Master 3.x uses API version 2, and the PuppetServer for Puppet 4.x uses API version 3. By default, API version 3 is used, so you only need to set this option if you are using Puppet Master 3.x. |
+| `--puppet-master HOSTNAME:PORT` | The hostname and port number of the Puppet Master. (By default the port used by Puppet Master is 8140.) |
+| `--puppet-master-api-version VERSION` | The API version used by the Puppet Master. API versions 2, 3,and 4 are supported. Puppet Master 3.x uses API version 2, and the PuppetServer for Puppet 4.x uses API version 3. PuppetServer 6.3.0 introduced the optional use of the v4 API but still fully supports the v3 API. By default, API version 3 is used, so you only need to set this option if you are using Puppet Master 3.x or wish to use the newer v4 API with PuppetServer 6. |
 | `--puppet-master-ssl-ca PATH` | Path to the CA certificate (public portion of certificate only) for your Puppet Master. This file will be on your Puppet Master and all Puppet agents. You can find it by running `puppet config print cacert` on any Puppet-managed host. |
-| `--puppet-master-ssl-client-cert PATH` | Path to the client certificate. Please see the section below on certificate authentication. |
-| `--puppet-master-ssl-client-key PATH` | Path to the client private key. Please see the section below on certificate authentication. |
+| `--puppet-master-ssl-client-cert PATH` | Path to the client certificate. Please see the section below on certificate authentication. This can be omitted if using PE RBAC token based auth with the v4 API. |
+| `--puppet-master-ssl-client-key PATH` | Path to the client private key. Please see the section below on certificate authentication. This can be omitted if using PE RBAC token based auth with the v4 API. |
+| `--puppet-master-token STRING` | A PE RBAC token used to authenticate a v4 catalog compile, in lieu of using certificate authentication. Please see the section below on token authentication. |
+| `--puppet-master-token-file PATH` | A path to a file containing a PE RBAC token used to authenticate a v4 catalog compile, in lieu of using certificate authentication. If this and `--puppet-master-token` are both specified, `--puppet-master-token` will be used instead. Please see the section below on token authentication. |
+| `--puppet-master-update-catalog` | When using the v4 API, instruct the PuppetServer to update the catalog generated from the compile in its PuppetDB instance. When using v2 and v3 APIs the catalog is always updated and this option is ignored. |
+| `--puppet-master-update-facts` | When using the v4 API, instruct the PuppetServer to update the facts used during the compile in its PuppetDB instance. When using v2 and v3 APIs the facts are always updated and this option is ignored. |
 
 If you wish to use a different Puppet Master to compile the "to" and "from" catalogs, you may prefix any of the `--puppet-master...` options with `to` or `from`. For example, perhaps you are testing an upgrade from Puppet 3.x to 4.x. You could use:
 
@@ -48,3 +56,13 @@ allow $1
 ```
 
 Please follow the instructions for the version of Puppet Master, PuppetServer, or Puppet Enterprise that you are using in order to generate and authorize the certificates.
+
+## PE RBAC Token authorization
+
+In newer versions of Puppet Enterprise you can authenticate using a valid PE RBAC token with appropriate permissions as long as it is authorized in the PuppetServer `auth.conf` file.
+
+By default this permission is enabled and controlled by the `puppet_enterprise::master::tk_authz::allow_rbac_catalog_compile` Hiera setting.
+
+The user the token was issued to must have the `puppetserver:compile_catalogs:*` permission.
+
+Note: A Puppet catalog may contain unencrypted secrets, even ones marked as `Sensitive`. In order to perform its job, Octocatalog-Diff needs access to the catalog. By granting a user the above RBAC permission you are granting them the ability to retrieve and view the complete catalog resulting from a compile, including any included secrets.

lib/octocatalog-diff/catalog.rb

@@ -191,6 +191,8 @@ def resources
       build
       raise OctocatalogDiff::Errors::CatalogError, 'Catalog does not appear to have been built' if !valid? && error_message.nil?
       raise OctocatalogDiff::Errors::CatalogError, error_message unless valid?
+      # Handle the structure returned by the /puppet/v4/catalog Puppetserver endpoint:
+      return @catalog['catalog']['resources'] if @catalog['catalog'].is_a?(Hash) && @catalog['catalog']['resources'].is_a?(Array)
       return @catalog['data']['resources'] if @catalog['data'].is_a?(Hash) && @catalog['data']['resources'].is_a?(Array)
       return @catalog['resources'] if @catalog['resources'].is_a?(Array)
       # This is a bug condition

lib/octocatalog-diff/catalog/puppetmaster.rb

@@ -62,16 +62,19 @@ def build_catalog(logger = Logger.new(StringIO.new))
         fetch_catalog(logger)
       end
 
-      # Returns a hash of parameters for each supported version of the Puppet Server Catalog API.
+      # Returns a hash of parameters for the requested version of the Puppet Server Catalog API.
       # @return [Hash] Hash of parameters
       #
       # Note: The double escaping of the facts here is implemented to correspond to a long standing
       # bug in the Puppet code. See https://github.com/puppetlabs/puppet/pull/1818 and
       # https://docs.puppet.com/puppet/latest/http_api/http_catalog.html#parameters for explanation.
-      def puppet_catalog_api
-        {
+      def puppet_catalog_api(version)
+        api_style = {
           2 => {
             url: "https://#{@options[:puppet_master]}/#{@options[:branch]}/catalog/#{@node}",
+            headers: {
+              'Accept' => 'text/pson'
+            },
             parameters: {
               'facts_format' => 'pson',
               'facts' => CGI.escape(@facts.fudge_timestamp.without('trusted').to_pson),
@@ -80,24 +83,59 @@ def puppet_catalog_api
           },
           3 => {
             url: "https://#{@options[:puppet_master]}/puppet/v3/catalog/#{@node}",
+            headers: {
+              'Accept' => 'text/pson'
+            },
             parameters: {
               'environment' => @options[:branch],
               'facts_format' => 'pson',
               'facts' => CGI.escape(@facts.fudge_timestamp.without('trusted').to_pson),
               'transaction_uuid' => SecureRandom.uuid
             }
+          },
+          4 => {
+            url: "https://#{@options[:puppet_master]}/puppet/v4/catalog",
+            headers: {
+              'Content-Type' => 'application/json'
+            },
+            parameters: {
+              'certname' => @node,
+              'persistence' => {
+                'facts' => @options[:puppet_master_update_facts] || false,
+                'catalog' => @options[:puppet_master_update_catalog] || false
+              },
+              'environment' => @options[:branch],
+              'facts' => { 'values' => @facts.facts['values'] },
+              'options' => {
+                'prefer_requested_environment' => true,
+                'capture_logs' => false,
+                'log_level' => 'warning'
+              },
+              'transaction_uuid' => SecureRandom.uuid
+            }
           }
         }
+
+        params = api_style[version]
+        return nil if params.nil?
+
+        unless @options[:puppet_master_token].nil?
+          params[:headers]['X-Authentication'] = @options[:puppet_master_token]
+        end
+
+        params[:parameters] = params[:parameters].to_json if version >= 4
+
+        params
       end
 
       # Fetch catalog by contacting the Puppet master, sending the facts, and asking for the catalog. When the
       # catalog is returned in PSON format, parse it to JSON and then set appropriate variables.
       def fetch_catalog(logger)
         api_version = @options[:puppet_master_api_version] || DEFAULT_PUPPET_SERVER_API
-        api = puppet_catalog_api[api_version]
+        api = puppet_catalog_api(api_version)
         raise ArgumentError, "Unsupported or invalid API version #{api_version}" unless api.is_a?(Hash)
 
-        more_options = { headers: { 'Accept' => 'text/pson' }, timeout: @timeout }
+        more_options = { headers: api[:headers], timeout: @timeout }
         post_hash = api[:parameters]
 
         response = nil

lib/octocatalog-diff/cli/options.rb

@@ -103,6 +103,7 @@ def self.option_globally_or_per_branch(opts = {})
         datatype = opts.fetch(:datatype, '')
         return option_globally_or_per_branch_string(opts) if datatype.is_a?(String)
         return option_globally_or_per_branch_array(opts) if datatype.is_a?(Array)
+        return option_globally_or_per_branch_boolean(opts) if datatype.is_a?(TrueClass) || datatype.is_a?(FalseClass)
         raise ArgumentError, "option_globally_or_per_branch not equipped to handle #{datatype.class}"
       end
 
@@ -177,6 +178,40 @@ def self.option_globally_or_per_branch_array(opts = {})
         end
       end
 
+      # See description of `option_globally_or_per_branch`. This implements the logic for a boolean value.
+      # @param :parser [OptionParser object] The OptionParser argument
+      # @param :options [Hash] Options hash being constructed; this is modified in this method.
+      # @param :cli_name [String] Name of option on command line (e.g. puppet-binary)
+      # @param :option_name [Symbol] Name of option in the options hash (e.g. :puppet_binary)
+      # @param :desc [String] Description of option on the command line; will have "for the XX branch" appended
+      def self.option_globally_or_per_branch_boolean(opts)
+        parser = opts.fetch(:parser)
+        options = opts.fetch(:options)
+        cli_name = opts.fetch(:cli_name)
+        option_name = opts.fetch(:option_name)
+        desc = opts.fetch(:desc)
+
+        flag = cli_name
+        from_option = "from_#{option_name}".to_sym
+        to_option = "to_#{option_name}".to_sym
+        parser.on("--[no-]#{flag}", "#{desc} globally") do |x|
+          translated = translate_option(opts[:translator], x)
+          options[to_option] = translated
+          options[from_option] = translated
+          post_process(opts[:post_process], options)
+        end
+        parser.on("--[no-]to-#{flag}", "#{desc} for the to branch") do |x|
+          translated = translate_option(opts[:translator], x)
+          options[to_option] = translated
+          post_process(opts[:post_process], options)
+        end
+        parser.on("--[no-]from-#{flag}", "#{desc} for the from branch") do |x|
+          translated = translate_option(opts[:translator], x)
+          options[from_option] = translated
+          post_process(opts[:post_process], options)
+        end
+      end
+
       # If a validator was provided, run the validator on the supplied value. The validator is expected to
       # throw an error if there is a problem. Note that the validator runs *before* the translator if both
       # a validator and translator are supplied.

lib/octocatalog-diff/cli/options/puppet_master_api_version.rb

@@ -14,8 +14,8 @@ def parse(parser, options)
       options: options,
       cli_name: 'puppet-master-api-version',
       option_name: 'puppet_master_api_version',
-      desc: 'Puppet Master API version (2 for Puppet 3.x, 3 for Puppet 4.x)',
-      validator: ->(x) { x =~ /^[23]$/ || raise(ArgumentError, 'Only API versions 2 and 3 are supported') },
+      desc: 'Puppet Master API version (2 for Puppet 3.x, 3 for Puppet 4.x, 4 for Puppet Server >= 6.3.0)',
+      validator: ->(x) { x =~ /^[234]$/ || raise(ArgumentError, 'Only API versions 2, 3, and 4 are supported') },
       translator: ->(x) { x.to_i }
     )
   end

lib/octocatalog-diff/cli/options/puppet_master_token.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+# Specify a PE RBAC token used to authenticate to Puppetserver for v4
+# catalog API calls.
+# @param parser [OptionParser object] The OptionParser argument
+# @param options [Hash] Options hash being constructed; this is modified in this method.
+OctocatalogDiff::Cli::Options::Option.newoption(:puppet_master_token) do
+  has_weight 310
+
+  def parse(parser, options)
+    OctocatalogDiff::Cli::Options.option_globally_or_per_branch(
+      parser: parser,
+      options: options,
+      datatype: '',
+      cli_name: 'puppet-master-token',
+      option_name: 'puppet_master_token',
+      desc: 'PE RBAC token to authenticate to the Puppetserver API v4'
+    )
+  end
+end

lib/octocatalog-diff/cli/options/puppet_master_token_file.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+# Specify a path to a file containing a PE RBAC token used to authenticate to the
+# Puppetserver for a v4 catalog API call.
+# @param parser [OptionParser object] The OptionParser argument
+# @param options [Hash] Options hash being constructed; this is modified in this method.
+OctocatalogDiff::Cli::Options::Option.newoption(:puppet_master_token_file) do
+  has_weight 300
+
+  def parse(parser, options)
+    OctocatalogDiff::Cli::Options.option_globally_or_per_branch(
+      parser: parser,
+      options: options,
+      datatype: '',
+      cli_name: 'puppet-master-token-file',
+      option_name: 'puppet_master_token_file',
+      desc: 'File containing PE RBAC token to authenticate to the Puppetserver API v4',
+      translator: ->(x) { x.start_with?('/', '~') ? x : File.join(options[:basedir], x) },
+      post_process: lambda do |opts|
+        %w(to from).each do |prefix|
+          fileopt = "#{prefix}_puppet_master_token_file".to_sym
+          tokenopt = "#{prefix}_puppet_master_token".to_sym
+
+          tokenfile = opts[fileopt]
+          next if tokenfile.nil?
+
+          raise(Errno::ENOENT, "Token file #{tokenfile} is not readable") unless File.readable?(tokenfile)
+
+          token = File.read(tokenfile).strip
+          opts[tokenopt] ||= token
+        end
+      end
+    )
+  end
+end

lib/octocatalog-diff/cli/options/puppet_master_update_catalog.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+# Specify if, when using the Puppetserver v4 catalog API, the Puppetserver should
+# update the catalog in PuppetDB.
+# @param parser [OptionParser object] The OptionParser argument
+# @param options [Hash] Options hash being constructed; this is modified in this method.
+OctocatalogDiff::Cli::Options::Option.newoption(:puppet_master_update_catalog) do
+  has_weight 320
+
+  def parse(parser, options)
+    OctocatalogDiff::Cli::Options.option_globally_or_per_branch(
+      parser: parser,
+      options: options,
+      datatype: false,
+      cli_name: 'puppet-master-update-catalog',
+      option_name: 'puppet_master_update_catalog',
+      desc: 'Update catalog in PuppetDB when using Puppetmaster API version 4'
+    )
+  end
+end

lib/octocatalog-diff/cli/options/puppet_master_update_facts.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+# Specify if, when using the Puppetserver v4 catalog API, the Puppetserver should
+# update the facts in PuppetDB.
+# @param parser [OptionParser object] The OptionParser argument
+# @param options [Hash] Options hash being constructed; this is modified in this method.
+OctocatalogDiff::Cli::Options::Option.newoption(:puppet_master_update_facts) do
+  has_weight 320
+
+  def parse(parser, options)
+    OctocatalogDiff::Cli::Options.option_globally_or_per_branch(
+      parser: parser,
+      options: options,
+      datatype: false,
+      cli_name: 'puppet-master-update-facts',
+      option_name: 'puppet_master_update_facts',
+      desc: 'Update facts in PuppetDB when using Puppetmaster API version 4'
+    )
+  end
+end

spec/octocatalog-diff/fixtures/catalogs/tiny-catalog-v4-api.json

@@ -0,0 +1,41 @@
+{
+  "catalog": {
+    "tags": ["settings"],
+    "name": "my.rspec.node",
+    "version": "production",
+    "code_id": null,
+    "catalog_uuid": "89869359-db50-472f-b435-1d37c22be9eb",
+    "catalog_format": 1,
+    "environment": "production",
+    "resources": [
+      {
+        "type": "Stage",
+        "title": "main",
+        "tags": ["stage"],
+        "exported": false,
+        "parameters": {
+          "name": "main"
+        }
+      },
+      {
+        "type": "Class",
+        "title": "Settings",
+        "tags": ["class","settings"],
+        "exported": false
+      }
+    ],
+    "edges": [
+      {
+        "source": "Stage[main]",
+        "target": "Class[Settings]"
+      },
+      {
+        "source": "Stage[main]",
+        "target": "Class[main]"
+      }
+    ],
+    "classes": [
+      "settings"
+    ]
+  }
+}

spec/octocatalog-diff/fixtures/configs/puppet-master-token.txt

@@ -0,0 +1 @@
+secretpuppetmastertoken