Skip to content

Commit 0a278ae

Browse files
docs: Document loading of custom components via git-sync
1 parent c2849e4 commit 0a278ae

File tree

2 files changed

+88
-3
lines changed

2 files changed

+88
-3
lines changed
Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
= Loading custom components
2+
:description: Load custom NiFi components for enhanced functionality.
3+
:nifi-docs-developers-guide: https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
4+
:nifi-docs-python-developers-guide: https://nifi.apache.org/nifi-docs/python-developer-guide.html
5+
:git-sync-docs: https://github.com/kubernetes/git-sync/tree/v4.2.4#manual
6+
7+
You can develop or use custom components for Apache NiFi, typically custom processors, to extend its functionality.
8+
9+
There are currently two types of custom components:
10+
11+
1. Custom NiFi Archives (NARs) +
12+
The {nifi-docs-developers-guide}[NiFi Developer’s Guide] provides further information on how to develop custom NARs.
13+
2. Starting with NiFi 2.0, also custom Python extensions can be used. +
14+
In {nifi-docs-python-developers-guide}, the development of custom components using Python is described.
15+
16+
The Stackable image contains the required tooling for both types.
17+
For instance, a supported Python version is included.
18+
NARs are only loaded once, but for Python scripts, hot-reloading is supported.
19+
20+
TIP: You might need to refresh your browser window to see the new or changed Python components.
21+
22+
Several options are described below, to add custom components to NiFi.
23+
24+
== git-sync
25+
26+
Custom NiFi components can be synchronized from a Git repository directly into the NiFi pods with git-sync.
27+
The {crd-docs}/nifi.stackable.tech/nificluster/v1alpha1[NifiCluster CRD] allows the specification of one or multiple Git repositories:
28+
29+
[source,yaml]
30+
----
31+
---
32+
apiVersion: nifi.stackable.tech/v1alpha1
33+
kind: NifiCluster
34+
spec:
35+
clusterConfig:
36+
customComponentsGitSync: # <1>
37+
- repo: https://example.com/git/custom-nifi-components # <2>
38+
branch: main # <3>
39+
gitFolder: path/to/the/components # <4>
40+
depth: 10 # <5>
41+
wait: 10s # <6>
42+
credentialsSecret: git-credentials # <7>
43+
gitSyncConf: # <8>
44+
--git-config: http.sslCAInfo:/tmp/ca-cert/ca.crt
45+
- repo: https://example.com/git/other-nifi-components # <9>
46+
nodes:
47+
config:
48+
logging:
49+
enableVectorAgent: true
50+
containers:
51+
git-sync: # <10>
52+
console:
53+
level: INFO
54+
file:
55+
level: INFO
56+
loggers:
57+
ROOT:
58+
level: INFO
59+
---
60+
apiVersion: v1
61+
kind: Secret
62+
metadata:
63+
name: git-credentials
64+
type: Opaque
65+
data:
66+
user: ...
67+
password: ...
68+
----
69+
<1> If the optional field `customComponentsGitSync` is defined, then containers running git-sync are deployed to synchronize the specified repositories.
70+
<2> The git repository URL that will be cloned
71+
<3> The branch to clone; defaults to `main`
72+
<4> Location in the Git repository containing the NiFi components; defaults to the root folder
73+
<5> The depth of synchronizing, i.e. the number of commits to clone; defaults to 1
74+
<6> The synchronization interval, e.g. `20s` or `5m`; defaults to `20s`
75+
<7> The name of the Secret used to access the repository if it is not public. +
76+
The referenced Secret must include two fields: `user` and `password`.
77+
The `password` field can either be an actual password (not recommended) or a GitHub token, as described in the {git-sync-docs}[git-sync documentation].
78+
<8> A map of optional configuration settings that are listed in the {git-sync-docs}[git-sync documentation]. +
79+
These settings are not verified.
80+
<9> Multiple repositories can be defined. Only the `repo` field is mandatory.
81+
<10> Logging can be configured as described in xref:concepts:logging.adoc[].
82+
As git-sync is a command-line tool, just its output is logged and no fine-grained log configuration is possible.
83+
All git-sync containers are configured via the one `git-sync` field.
84+
85+
It cannot be specified, if a repository contains NiFi Archives or Python components.
86+
The operator just configures each repository for both types.
87+
In particular, the parameters `nifi.nar.library.directory` and `nifi.python.extensions.source.directory` are set.

docs/modules/nifi/partials/nav.adoc

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,7 @@
1616
** xref:nifi:usage_guide/exposing-processors/index.adoc[]
1717
*** xref:nifi:usage_guide/exposing-processors/http.adoc[]
1818
*** xref:nifi:usage_guide/exposing-processors/tcp.adoc[]
19-
** xref:nifi:usage_guide/custom-components/index.adoc[]
20-
*** xref:nifi:usage_guide/custom-components/custom-nars.adoc[]
21-
*** xref:nifi:usage_guide/custom-components/custom-python-processors.adoc[]
19+
** xref:nifi:usage_guide/custom-components.adoc[]
2220
** xref:nifi:usage_guide/operations/index.adoc[]
2321
*** xref:nifi:usage_guide/operations/cluster-operations.adoc[]
2422
*** xref:nifi:usage_guide/operations/pod-placement.adoc[]

0 commit comments

Comments
 (0)