describe and implement the external protocol

.spellcheck-en-custom.txt

@@ -30,3 +30,5 @@ kickstarts
 koji
 livemediacreator
 untrusted
+executables
+subtree

doc/03-omnifest/01-directive.md

@@ -199,8 +199,8 @@ otk.meta.kiwi:
 ## `otk.external`
 
 External directives. Directives starting with `otk.external` are redirected
-to `/usr/libexec/otk/`-binaries. For example the directive
-`otk.external.osbuild.depsolve_dnf4` will execute `otk-osbuild depsolve_dnf4`
+to `/usr/libexec/otk/external`-binaries. For example the directive
+`otk.external.osbuild_depsolve_dnf4` will execute `osbuild_depsolve_dnf4`
 with the tree under the directive on stdin and expect a new tree to replace
 the directive with on stdout.
 

doc/03-omnifest/02-external.md

@@ -1,22 +1,173 @@
 # External
 
-## osbuild
+External directives are directives that are implemented externally from `otk`.
+They are meant to be used to provide target-specific behavior and can be used
+to express things where `otk` is not expressive enough.
+
+## Protocol
+
+Directives are small executables that receive JSON and are expected to output
+JSON again in a specific format. Additional keys are not allowed.
+
+When we have an [omnifest](./index.md) that looks like this:
+
+```yaml
+otk.external.name:
+  child:
+    - 1
+  options: "are here"
+```
+
+The external gets called with the following JSON:
+
+```json
+{
+  "tree": {
+    "otk.external.name": {
+      "child": [1],
+      "options": "are here"
+    }
+  }
+}
+```
+
+And is expected to return a JSON dict with a single top-level `tree` object that can contain arbitrary values. Additional top-level keys are not allowed. Example:
+
+```json
+{
+  "tree": {
+    "what": {
+      "you": "want"
+    }
+  }
+}
+```
+
+Which will replace the previous `otk.external.name` subtree with the output of new subtree from the external command. Example:
+
+```yaml
+tree:
+  what:
+    you: "want"
+```
+
+If the external returns `{}`, an empty object, `otk` will assume that there is
+no tree to replace and remove the node instead. This will turn the following
+YAML:
+
+```yaml
+dummy:
+  otk.external.name:
+    options:
+```
+
+Into this YAML structure:
+
+```yaml
+dummy:
+```
+
+## Example
+
+The following example demonstrates how an external can be used to implement string
+concatenation.
+
+Given the following script called `concat` in `/usr/local/libexec/otk/concat`:
+
+```bash
+#!/usr/bin/env bash
+
+output="$(jq -jr '.tree."otk.external.concat".parts[]' <<< "${1}")"
+echo "{\"tree\":{\"output\":\"$output\"}}"
+```
+
+and the following directive:
+
+```yaml
+examplestring:
+  otk.external.concat:
+    parts:
+      - list
+      - of
+      - strings
+```
+
+the script is called with the following data on stdin:
+
+```json
+{
+  "tree": {
+    "otk.external.concat": {
+      "parts": [
+        "list",
+        "of",
+        "strings"
+      ]
+    }
+  }
+}
+```
+
+which results in the following output:
+
+```json
+{
+  "tree": {
+    "string": "listofstrings"
+  }
+}
+```
+
+and the final omnifest will be:
+
+```yaml
+examplestring:
+  listofstrings
+```
+
+## Paths
+
+`otk` will look for external directives in the following paths, stopping when
+it finds the first match:
+
+- A path defined by the `OTK_EXTERNAL_PATH` environment variable.
+- `/usr/local/libexec/otk/external`
+- `/usr/libexec/otk/external`
+- `/usr/local/lib/otk/external`
+- `/usr/lib/otk/external`
+
+The filename for an external executable is based on the external name. When the
+following directive is encountered: `otk.external.<name>` then
+`otk` will try to find an executable called `<name>` in the previously
+mentioned search paths.
+
+Examples:
+
+- `otk.external.foo` -> `foo`
+- `otk.external.osbuild_bar` -> `osbuild_bar`
+
+## Implementations
+
+`otk` currently ships together with an external implementation for `osbuild`.
+These directives can be used as children under an `otk.target.osbuild` tree.
+
+### osbuild
 
 These directives are only allowed within a [`otk.target.osbuild.<name>`](./01-directive.md#otktargetconsumername).
 
-### `otk.external.osbuild.depsolve_dnf4`
+#### `otk.external.osbuild_depsolve_dnf4`
 
 Solves a list of package specifications to RPMs and specifies them in the
 osbuild manifest as sources.
 
-### `otk.external.osbuild.depsolve_dnf5`
+#### `otk.external.osbuild_depsolve_dnf5`
 
 Expects a `map` as its value.
 
 `osbuild` directives to write files. **If a stage exists for the type of file
 you want to write: use it.** See the [best practices](../04-best-practices.md).
 
-### `otk.external.osbuild.file_from_text`
+#### `otk.external.osbuild_file_from_text`
 
 Write inline text to a file. Creates a source in the manifest and copies that
 source to the destination in the tree.
@@ -30,15 +181,15 @@ otk.external.osbuild.file_from_text:
     Hello, World!
 ```
 
-### `otk.external.osbuild.file_from_path`
+#### `otk.external.osbuild_file_from_path`
 
 Copy a file. Source is relative to the path of the entrypoint omnifest. Creates
 a source in the manifest and copies that source to the destination in tree.
 
 Path components up to the destination must be pre-existing in the tree.
 
 ```yaml
-otk.external.osbuild.file_from_path:
+otk.external.osbuild_file_from_path:
   source: README.md
   destination: /path/to
 ```

example/centos/pipeline/build.yaml

@@ -4,7 +4,7 @@ name: build
 source-epoch: ${source_epoch}
 runner: "org.osbuild.centos9"
 stages:
-  - otk.external.osbuild.depsolve_dnf4:
+  - otk.external.osbuild_depsolve_dnf4:
     architecture: ${architecture}
     releasever: ${version}
     module_platform_id: platform:el${version}

example/centos/pipeline/os-ostree.yaml

@@ -3,7 +3,7 @@ build: "name:build"
 
 stages:
   # Install RPMs
-  - otk.external.osbuild.depsolve_dnf4:
+  - otk.external.osbuild_depsolve_dnf4:
     architecture: ${architecture}
     releasever: ${version}
     module_platform_id: platform:el${version}

example/centos/pipeline/os.yaml

@@ -3,7 +3,7 @@ build: "name:build"
 
 stages:
   # Install RPMs
-  - otk.external.osbuild.depsolve_dnf4:
+  - otk.external.osbuild_depsolve_dnf4:
     architecture: ${architecture}
     releasever: ${version}
     module_platform_id: platform:el${version}

example/fedora/osbuild/buildroot.yaml

@@ -2,7 +2,7 @@
 name: buildroot
 source-epoch: 1659397331
 stages:
-  - otk.external.osbuild.depsolve_dnf4:
+  - otk.external.osbuild_depsolve_dnf4:
       architecture: ${architecture}
       releasever: "40"
       module_platform_id: f${version}

example/fedora/osbuild/pipeline/raw.yaml

@@ -1,7 +1,7 @@
 name: raw
 source-epoch: 1659397331
 stages:
-  #  - otk.external.osbuild.depsolve_dnf4:
+  #  - otk.external.osbuild_depsolve_dnf4:
   #      architecture: ${architecture}
   #      module_platform_id: f${version}
   #      repositories: ${repositories}

example/fedora/osbuild/pipeline/tree.yaml

@@ -1,7 +1,7 @@
 name: tree
 build: name:root
 stages:
-  - otk.external.osbuild.depsolve_dnf4:
+  - otk.external.osbuild_depsolve_dnf4:
       architecture: ${architecture}
       releasever: "40"
       module_platform_id: f${version}

otk.spec

@@ -43,7 +43,7 @@ cp -a example/* %{buildroot}%{_datadir}/otk/example/
 %doc doc
 %{_datadir}/otk
 %{_bindir}/otk
-%{_bindir}/otk-osbuild
+%{_bindir}/otk_external_osbuild
 %{python3_sitelib}/otk
 %{python3_sitelib}/otk_osbuild
 %{python3_sitelib}/otk-%{version}.dist-info

pyproject.toml

@@ -10,7 +10,7 @@ dependencies = [
 
 [project.scripts]
 otk = "otk.command:root"
-otk-osbuild = "otk_osbuild.command:root"
+otk_external_osbuild = "otk_external_osbuild.command:root"
 
 [project.optional-dependencies]
 dev = [

src/otk/constant.py

@@ -6,4 +6,6 @@
 PREFIX_INCLUDE = f"{PREFIX}include"
 PREFIX_DEFINE = f"{PREFIX}define"
 
+PREFIX_EXTERNAL = f"{PREFIX}external."
+
 NAME_VERSION = f"{PREFIX}version"