Updating find_unused_modules script

[apinnick]

What changes are you introducing?

Added a line to detect snippets included in modules.

Added a comment that script does not work on file includes that contain attributes. User will have to ignore those files in the output.

Why are you introducing these changes? (Explanation, links to references, issues, etc.)

Snippets included in modules were reported as false positives.

Anything else to add? (Considerations, potential downsides, alternative solutions you have explored, etc.)

I tested this with a fake snippet. It was detected.

No style review required.

Checklists

• I am okay with my commits getting squashed when you merge this PR.
• I am familiar with the contributing guidelines.

Please cherry-pick my commits into:

• Foreman 3.14/Katello 4.16
• Foreman 3.13/Katello 4.15 (EL9 only)
• Foreman 3.12/Katello 4.14 (Satellite 6.16)
• Foreman 3.11/Katello 4.13 (orcharhino 6.11 on EL8 only; orcharhino 7.0 on EL8+EL9; orcharhino 7.1 with Leapp)
• Foreman 3.10/Katello 4.12
• Foreman 3.9/Katello 4.11 (Satellite 6.15; orcharhino 6.8/6.9/6.10)
• Foreman 3.8/Katello 4.10
• Foreman 3.7/Katello 4.9 (Satellite 6.14)
• We do not accept PRs for Foreman older than 3.7.

[github-actions]

The PR preview for c4b9779 is available at theforeman-foreman-documentation-preview-pr-3760.surge.sh

No diff compared to the current base

show diff

[maximiliankolb]

Tested locally; it works as expected:

$ ./guides/scripts/find_unused_modules.py
proc_configuring-repositories-katello.adoc
proc_configuring-repositories-foreman-el.adoc
proc_configuring-repositories-satellite.adoc
proc_configuring-repositories-orcharhino.adoc
$ echo $?
0

There are still false positives but this is much much better than before. Thanks Avital.

[ekohl]

Something that I came across when playing with #3741 was the :sourcemap option in https://docs.asciidoctor.org/asciidoctor/latest/api/options/

Tracks the file and line number for each parsed block. Useful for tooling applications where the association between the converted output and the source file is important.

https://docs.asciidoctor.org/asciidoctor/latest/api/sourcemap/ is a longer document that shows asciidoctor internally keeps track of all the files it uses. It actually made me think about creating debug outputs where we display within the rendered guide the actual source location for each paragraph.

Taking it back to this script: would it make sense to adopt a strategy of actually rendering all guides in all flavors and then look at which files it actually used? That way we're not guessing. It will also catch edge cases where we use variables in filenames.

[apinnick]

Something that I came across when playing with #3741 was the :sourcemap option in https://docs.asciidoctor.org/asciidoctor/latest/api/options/

Tracks the file and line number for each parsed block. Useful for tooling applications where the association between the converted output and the source file is important.

https://docs.asciidoctor.org/asciidoctor/latest/api/sourcemap/ is a longer document that shows asciidoctor internally keeps track of all the files it uses. It actually made me think about creating debug outputs where we display within the rendered guide the actual source location for each paragraph.

Taking it back to this script: would it make sense to adopt a strategy of actually rendering all guides in all flavors and then look at which files it actually used? That way we're not guessing. It will also catch edge cases where we use variables in filenames.

@ekohl Pity we didn't know about this while we were still committed to Asciidoc! 😄

At the moment, it's probably best to keep our tools as generic as possible. The effects of the d/s tooling changes are still a mystery. When we know more, we can think about updating our tools.

[ekohl]

I couldn't help myself:

#!/usr/bin/env ruby

# frozen_string_literal: true

require 'asciidoctor'

BUILDS = %w[
  foreman-el
  foreman-deb
  katello
  satellite
  orcharhino
].freeze

BASE_DIR = File.dirname(__dir__)

included = Set.new

Dir.glob(File.join(BASE_DIR, '*', 'master.adoc')).each do |path|
  BUILDS.each do |build|
    doc = Asciidoctor.load_file(
      path,
      doctype: :book,
      safe: :safe,
      base_dir: BASE_DIR,
      attributes: { 'build' => build }
    )

    included += doc.catalog[:includes].keys
  end
end

all_modules = Dir.glob(File.join(BASE_DIR, 'common', 'modules', '**', '*.adoc'))
absolute_included = included.map { |path| File.join(BASE_DIR, "#{path}.adoc") }

puts all_modules - absolute_included

$ guides/scripts/find_modules
asciidoctor: ERROR: master.adoc: line 5: include file not found: /home/ekohl/dev/foreman-documentation/guides/topics/foreman.adoc
asciidoctor: ERROR: master.adoc: line 21: include file not found: /home/ekohl/dev/foreman-documentation/guides/topics/foreman-contributors.adoc
asciidoctor: ERROR: master.adoc: line 5: include file not found: /home/ekohl/dev/foreman-documentation/guides/topics/foreman.adoc
asciidoctor: ERROR: master.adoc: line 21: include file not found: /home/ekohl/dev/foreman-documentation/guides/topics/foreman-contributors.adoc
asciidoctor: ERROR: master.adoc: line 11: include file not found: /home/ekohl/dev/foreman-documentation/guides/topics/foreman.adoc
asciidoctor: ERROR: master.adoc: line 12: include file not found: /home/ekohl/dev/foreman-documentation/guides/topics/katello.adoc
asciidoctor: ERROR: master.adoc: line 21: include file not found: /home/ekohl/dev/foreman-documentation/guides/topics/foreman-contributors.adoc
asciidoctor: ERROR: master.adoc: line 28: include file not found: /home/ekohl/dev/foreman-documentation/guides/topics/katello-contributors.adoc
asciidoctor: ERROR: master.adoc: line 5: include file not found: /home/ekohl/dev/foreman-documentation/guides/topics/foreman.adoc
asciidoctor: ERROR: master.adoc: line 21: include file not found: /home/ekohl/dev/foreman-documentation/guides/topics/foreman-contributors.adoc
asciidoctor: ERROR: master.adoc: line 5: include file not found: /home/ekohl/dev/foreman-documentation/guides/topics/foreman.adoc
asciidoctor: ERROR: master.adoc: line 21: include file not found: /home/ekohl/dev/foreman-documentation/guides/topics/foreman-contributors.adoc
/home/ekohl/dev/foreman-documentation/guides/common/modules/con_configuring-keycloak-wildfly-authentication-with-piv-cards-for-project.adoc
/home/ekohl/dev/foreman-documentation/guides/common/modules/snip_prerequisites-content-for-sles.adoc

This is a different approach. I'm ignoring the errors from the release notes for now.

It shows a false positive for snip_prerequisites-content-for-sles.adoc, probably because the way we render orcharhino isn't quite correct.

Then it finds con_configuring-keycloak-wildfly-authentication-with-piv-cards-for-project.adoc which is unused. It's included in assembly_configuring-keycloak-wildfly-authentication-with-piv-cards-for-project.adoc but that whole assembly itself appears to be unused.

[ekohl]

This is a different approach. I'm ignoring the errors from the release notes for now.

That's because I'm setting the base_dir incorrectly:

      base_dir: BASE_DIR,

This should be base_dir: File.dirname(path).

The whole file:

#!/usr/bin/env ruby

# frozen_string_literal: true

require 'asciidoctor'

BUILDS = %w[
  foreman-el
  foreman-deb
  katello
  satellite
  orcharhino
].freeze

BASE_DIR = File.dirname(__dir__)

included = Set.new

Dir.glob(File.join(BASE_DIR, '*', 'master.adoc')).each do |path|
  BUILDS.each do |build|
    doc = Asciidoctor.load_file(
      path,
      doctype: :book,
      safe: :safe,
      base_dir: File.dirname(path),
      attributes: { 'build' => build }
    )

    included += doc.catalog[:includes].keys
  end
end

all_modules = Dir.glob(File.join(BASE_DIR, 'common', 'modules', '**', '*.adoc'))
absolute_included = included.map { |path| File.join(BASE_DIR, "#{path}.adoc") }

puts all_modules - absolute_included

Now the output is:

$ guides/scripts/find_modules 
/home/ekohl/dev/foreman-documentation/guides/common/modules/con_configuring-keycloak-wildfly-authentication-with-piv-cards-for-project.adoc
/home/ekohl/dev/foreman-documentation/guides/common/modules/snip_prerequisites-content-for-sles.adoc