From 81d1441035b2ef334854a2858c72a1ebeea69608 Mon Sep 17 00:00:00 2001 From: Nicolas Ruflin Date: Mon, 22 Sep 2025 12:45:42 +0200 Subject: [PATCH 1/3] feat: add custom Kibana configuration support - Add support for custom kibana-custom.yml configuration files - Custom configs are appended to base kibana.yml configuration - Enable via stack.kibana_custom_config_enabled profile setting - Support template variables in custom configurations - Add comprehensive documentation and examples - Maintain backward compatibility with existing setups Resolves: Custom Kibana configuration enhancement --- README.md | 4 + docs/howto/custom_kibana_config.md | 191 ++++++++++++++++++++ internal/profile/_static/config.yml.example | 4 + internal/stack/kibana_config.go | 111 ++++++++++++ internal/stack/resources.go | 34 +++- 5 files changed, 336 insertions(+), 8 deletions(-) create mode 100644 docs/howto/custom_kibana_config.md create mode 100644 internal/stack/kibana_config.go diff --git a/README.md b/README.md index 0dbdab90a4..6edab25635 100644 --- a/README.md +++ b/README.md @@ -689,6 +689,10 @@ The following settings are available per profile: * `stack.elastic_subscription` allows to select the Elastic subscription type to be used in the stack. Currently, it is supported "basic" and "[trial](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-trial.html)", which enables all subscription features for 30 days. Defaults to "trial". +* `stack.kibana_custom_config_enabled` can be set to true to enable custom Kibana configuration. + When enabled, you can create a `kibana-custom.yml` file in your profile directory with additional + Kibana settings that will be appended to the base configuration. Defaults to false. + See [Custom Kibana Configuration](docs/howto/custom_kibana_config.md) for details. ## Useful environment variables diff --git a/docs/howto/custom_kibana_config.md b/docs/howto/custom_kibana_config.md new file mode 100644 index 0000000000..6f6f990127 --- /dev/null +++ b/docs/howto/custom_kibana_config.md @@ -0,0 +1,191 @@ +# Custom Kibana Configuration + +This document explains how to add custom configuration to Kibana when using elastic-package. + +## Overview + +You can provide additional Kibana configuration that will be appended to the base configuration generated by elastic-package. This allows you to: + +- Override default settings +- Add custom plugins configuration +- Set up custom security settings +- Configure additional features + +## Setup + +1. Enable custom configuration in your profile: + ```bash + # Edit ~/.elastic-package/profiles/default/config.yml + stack.kibana_custom_config_enabled: true + ``` + +2. Create your custom configuration file: + ```bash + # Create ~/.elastic-package/profiles/default/kibana-custom.yml + touch ~/.elastic-package/profiles/default/kibana-custom.yml + ``` + +3. Add your custom configuration: + ```yaml + # Example custom configuration + logging.loggers: + - name: plugins.security + level: debug + + server.customResponseHeaders: + X-Custom-Header: "MyValue" + + # Template variables are supported + xpack.security.enabled: {{ if eq .security_enabled "true" }}true{{ else }}false{{ end }} + ``` + +## Template Support + +Your custom configuration supports the same templating as the base configuration: + +- `{{ fact "kibana_version" }}` - Kibana version +- `{{ fact "username" }}` - Elasticsearch username +- `{{ fact "password" }}` - Elasticsearch password +- `{{ fact "apm_enabled" }}` - APM enabled flag +- `{{ fact "self_monitor_enabled" }}` - Self monitoring enabled flag +- All other facts available in the base template + +### Available Template Variables + +The following template variables are available in your custom configuration: + +| Variable | Description | Example | +|----------|-------------|---------| +| `kibana_version` | Version of Kibana | `8.11.0` | +| `elasticsearch_version` | Version of Elasticsearch | `8.11.0` | +| `agent_version` | Version of Elastic Agent | `8.11.0` | +| `username` | Elasticsearch username | `elastic` | +| `password` | Elasticsearch password | `changeme` | +| `kibana_host` | Kibana host URL | `https://kibana:5601` | +| `elasticsearch_host` | Elasticsearch host URL | `https://elasticsearch:9200` | +| `fleet_url` | Fleet server URL | `https://fleet-server:8220` | +| `apm_enabled` | APM enabled flag | `true`/`false` | +| `logstash_enabled` | Logstash enabled flag | `true`/`false` | +| `self_monitor_enabled` | Self monitoring flag | `true`/`false` | + +## Configuration Precedence + +1. Base elastic-package configuration (from template) +2. Your custom configuration (appended) + +Since YAML allows duplicate keys and Kibana processes them in order, your custom settings will override base settings with the same key. + +## Examples + +### Enable Debug Logging +```yaml +logging.loggers: + - name: root + level: debug + - name: plugins.fleet + level: trace +``` + +### Custom Security Settings +```yaml +xpack.security.session.idleTimeout: "8h" +xpack.security.session.lifespan: "24h" +xpack.security.authc.providers: + basic.basic1: + order: 0 +``` + +### Development Features +```yaml +server.rewriteBasePath: false +server.dev.basePathProxyTarget: 3000 + +# Enable experimental features +xpack.fleet.enableExperimental: + - customIntegrations + - agentTamperProtectionEnabled +``` + +### Custom UI Settings +```yaml +uiSettings: + overrides: + "theme:darkMode": true + "dateFormat": "YYYY-MM-DD HH:mm:ss.SSS" + "discover:sampleSize": 1000 +``` + +### Template Usage Example +```yaml +# Use template variables in your custom config +server.name: kibana-{{ fact "kibana_version" }} +server.customResponseHeaders: + X-Kibana-Version: "{{ fact "kibana_version" }}" + +# Conditional configuration based on features +{{ if eq (fact "apm_enabled") "true" }} +elastic.apm.active: true +elastic.apm.serverUrl: "http://fleet-server:8200" +elastic.apm.environment: "development" +{{ end }} + +# Version-specific configuration +{{ if semverLessThan (fact "kibana_version") "8.0.0" }} +# Legacy configuration for older versions +xpack.monitoring.ui.container.elasticsearch.enabled: true +{{ else }} +# Modern configuration for newer versions +monitoring.ui.container.elasticsearch.enabled: true +{{ end }} +``` + +## Troubleshooting + +### Configuration Not Applied +- Ensure `stack.kibana_custom_config_enabled: true` is set in your profile's `config.yml` +- Verify the `kibana-custom.yml` file exists in your profile directory +- Check that the YAML syntax is valid + +### Template Errors +- Use `{{ fact "variable_name" }}` syntax for template variables +- Ensure template functions like `semverLessThan` are used correctly +- Check the elastic-package logs for template parsing errors + +### Kibana Startup Issues +- Validate your custom configuration against Kibana documentation +- Start with minimal changes and add complexity gradually +- Check Kibana logs for configuration validation errors + +## Profile-Specific Configurations + +You can have different custom configurations for different profiles: + +```bash +# Development profile with debug settings +~/.elastic-package/profiles/development/kibana-custom.yml + +# Production profile with optimized settings +~/.elastic-package/profiles/production/kibana-custom.yml +``` + +Each profile can enable or disable custom configuration independently. + +## Best Practices + +1. **Start Simple**: Begin with basic configuration changes and add complexity gradually +2. **Use Comments**: Document your custom configurations for future reference +3. **Test Changes**: Verify that Kibana starts successfully after configuration changes +4. **Version Compatibility**: Use template conditions for version-specific settings +5. **Backup Configurations**: Keep copies of working configurations before making changes + +## Limitations + +- Custom configuration is appended to the base configuration (not merged at the YAML structure level) +- Template processing errors will prevent stack startup +- Some Kibana settings may require specific ordering or dependencies + +## Related Documentation + +- [Kibana Configuration Settings](https://www.elastic.co/guide/en/kibana/current/settings.html) +- [elastic-package Profiles](../README.md#profiles) +- [Stack Configuration](../README.md#stack-configuration) diff --git a/internal/profile/_static/config.yml.example b/internal/profile/_static/config.yml.example index d9f00f00a8..e81f99fe2e 100644 --- a/internal/profile/_static/config.yml.example +++ b/internal/profile/_static/config.yml.example @@ -29,3 +29,7 @@ ## Set license subscription # stack.elastic_subscription: "basic" + +## Custom Kibana Configuration +# Enable custom kibana.yml configuration file +# stack.kibana_custom_config_enabled: true diff --git a/internal/stack/kibana_config.go b/internal/stack/kibana_config.go new file mode 100644 index 0000000000..50b23c42d4 --- /dev/null +++ b/internal/stack/kibana_config.go @@ -0,0 +1,111 @@ +// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one +// or more contributor license agreements. Licensed under the Elastic License; +// you may not use this file except in compliance with the Elastic License. + +package stack + +import ( + "bytes" + "fmt" + "html/template" + "io" + "os" + + "github.com/elastic/go-resource" + + "github.com/elastic/elastic-package/internal/profile" +) + +// kibanaConfigWithCustomContent generates kibana.yml with custom config appended +func kibanaConfigWithCustomContent(profile *profile.Profile) func(resource.Context, io.Writer) error { + return func(ctx resource.Context, w io.Writer) error { + // First, generate the base kibana.yml from template + var baseConfig bytes.Buffer + baseTemplate := staticSource.Template("_static/kibana.yml.tmpl") + err := baseTemplate(ctx, &baseConfig) + if err != nil { + return fmt.Errorf("failed to generate base kibana config: %w", err) + } + + // Write base config to output + _, err = w.Write(baseConfig.Bytes()) + if err != nil { + return fmt.Errorf("failed to write base kibana config: %w", err) + } + + // Check if custom config is enabled and exists + if profile.Config(configKibanaCustomConfigEnabled, "false") == "false" { + return nil // No custom config needed + } + + customConfigPath := profile.Path(KibanaCustomConfigFile) + customConfigData, err := os.ReadFile(customConfigPath) + if os.IsNotExist(err) { + return nil // No custom config file, that's fine + } + if err != nil { + return fmt.Errorf("failed to read custom kibana config: %w", err) + } + + // Add separator comment + _, err = w.Write([]byte("\n\n# Custom Kibana Configuration\n")) + if err != nil { + return fmt.Errorf("failed to write custom config separator: %w", err) + } + + // Process custom config as template + customTemplate, err := template.New("kibana-custom"). + Funcs(templateFuncs). + Parse(string(customConfigData)) + if err != nil { + return fmt.Errorf("failed to parse custom kibana config template: %w", err) + } + + // Create template data from resource context facts + templateData := createTemplateDataFromContext(ctx) + + err = customTemplate.Execute(w, templateData) + if err != nil { + return fmt.Errorf("failed to execute custom kibana config template: %w", err) + } + + return nil + } +} + +// createTemplateDataFromContext creates template data from resource context +// This function extracts commonly used facts and makes them available to templates +func createTemplateDataFromContext(ctx resource.Context) map[string]interface{} { + data := make(map[string]interface{}) + + // List of facts that should be available in custom templates + factNames := []string{ + "kibana_version", + "elasticsearch_version", + "agent_version", + "username", + "password", + "kibana_host", + "elasticsearch_host", + "fleet_url", + "apm_enabled", + "logstash_enabled", + "self_monitor_enabled", + "kibana_http2_enabled", + "logsdb_enabled", + "elastic_subscription", + "geoip_dir", + "agent_publish_ports", + "api_key", + "enrollment_token", + } + + // Extract facts from context + for _, factName := range factNames { + if value, found := ctx.Fact(factName); found { + data[factName] = value + } + } + + return data +} diff --git a/internal/stack/resources.go b/internal/stack/resources.go index a16de5a4d9..8c90c7c711 100644 --- a/internal/stack/resources.go +++ b/internal/stack/resources.go @@ -34,6 +34,9 @@ const ( // KibanaConfigFile is the kibana config file. KibanaConfigFile = "kibana.yml" + // KibanaCustomConfigFile is the custom kibana config file. + KibanaCustomConfigFile = "kibana-custom.yml" + // LogstashConfigFile is the logstash config file. LogstashConfigFile = "logstash.conf" @@ -58,13 +61,14 @@ const ( elasticsearchUsername = "elastic" elasticsearchPassword = "changeme" - configAPMEnabled = "stack.apm_enabled" - configGeoIPDir = "stack.geoip_dir" - configKibanaHTTP2Enabled = "stack.kibana_http2_enabled" - configLogsDBEnabled = "stack.logsdb_enabled" - configLogstashEnabled = "stack.logstash_enabled" - configSelfMonitorEnabled = "stack.self_monitor_enabled" - configElasticSubscription = "stack.elastic_subscription" + configAPMEnabled = "stack.apm_enabled" + configGeoIPDir = "stack.geoip_dir" + configKibanaHTTP2Enabled = "stack.kibana_http2_enabled" + configKibanaCustomConfigEnabled = "stack.kibana_custom_config_enabled" + configLogsDBEnabled = "stack.logsdb_enabled" + configLogstashEnabled = "stack.logstash_enabled" + configSelfMonitorEnabled = "stack.self_monitor_enabled" + configElasticSubscription = "stack.elastic_subscription" ) var ( @@ -189,7 +193,21 @@ func applyResources(profile *profile.Profile, stackVersion string) error { resourceManager.RegisterProvider("file", &resource.FileProvider{ Prefix: stackDir, }) - resources := append([]resource.Resource{}, stackResources...) + // Create kibana resource with custom config support + kibanaResource := &resource.File{ + Path: KibanaConfigFile, + Content: kibanaConfigWithCustomContent(profile), + } + + // Replace the kibana resource in stackResources with our custom one + resources := make([]resource.Resource, 0, len(stackResources)) + for _, res := range stackResources { + if file, ok := res.(*resource.File); ok && file.Path == KibanaConfigFile { + resources = append(resources, kibanaResource) + } else { + resources = append(resources, res) + } + } // Keeping certificates in the profile directory for backwards compatibility reasons. resourceManager.RegisterProvider(CertsFolder, &resource.FileProvider{ From 1349eb61a7e67eba1f8e50103ae90d4163f95703 Mon Sep 17 00:00:00 2001 From: Nicolas Ruflin Date: Mon, 22 Sep 2025 15:31:43 +0200 Subject: [PATCH 2/3] refactor: simplify custom Kibana config to auto-detect file - Remove configKibanaCustomConfigEnabled option - Automatically detect kibana-custom.yml file existence - Simplify setup - just create the file to enable custom config - Update documentation to reflect simpler approach - Maintain backward compatibility and all existing functionality This makes the feature more discoverable and easier to use without requiring users to enable it via configuration. --- README.md | 11 +++++++---- docs/howto/custom_kibana_config.md | 14 ++++---------- internal/profile/_static/config.yml.example | 4 ---- internal/stack/kibana_config.go | 6 +----- internal/stack/resources.go | 15 +++++++-------- 5 files changed, 19 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index 6edab25635..08ba7bfd05 100644 --- a/README.md +++ b/README.md @@ -689,10 +689,13 @@ The following settings are available per profile: * `stack.elastic_subscription` allows to select the Elastic subscription type to be used in the stack. Currently, it is supported "basic" and "[trial](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-trial.html)", which enables all subscription features for 30 days. Defaults to "trial". -* `stack.kibana_custom_config_enabled` can be set to true to enable custom Kibana configuration. - When enabled, you can create a `kibana-custom.yml` file in your profile directory with additional - Kibana settings that will be appended to the base configuration. Defaults to false. - See [Custom Kibana Configuration](docs/howto/custom_kibana_config.md) for details. + +## Custom Kibana Configuration + +You can provide custom Kibana configuration by creating a `kibana-custom.yml` file in your profile directory. +The custom configuration will be automatically appended to the base configuration when the file exists. + +See [Custom Kibana Configuration](docs/howto/custom_kibana_config.md) for detailed instructions. ## Useful environment variables diff --git a/docs/howto/custom_kibana_config.md b/docs/howto/custom_kibana_config.md index 6f6f990127..17b87d6759 100644 --- a/docs/howto/custom_kibana_config.md +++ b/docs/howto/custom_kibana_config.md @@ -13,19 +13,13 @@ You can provide additional Kibana configuration that will be appended to the bas ## Setup -1. Enable custom configuration in your profile: - ```bash - # Edit ~/.elastic-package/profiles/default/config.yml - stack.kibana_custom_config_enabled: true - ``` - -2. Create your custom configuration file: +1. Create your custom configuration file: ```bash # Create ~/.elastic-package/profiles/default/kibana-custom.yml touch ~/.elastic-package/profiles/default/kibana-custom.yml ``` -3. Add your custom configuration: +2. Add your custom configuration: ```yaml # Example custom configuration logging.loggers: @@ -142,9 +136,9 @@ monitoring.ui.container.elasticsearch.enabled: true ## Troubleshooting ### Configuration Not Applied -- Ensure `stack.kibana_custom_config_enabled: true` is set in your profile's `config.yml` - Verify the `kibana-custom.yml` file exists in your profile directory - Check that the YAML syntax is valid +- Ensure the file is in the correct location: `~/.elastic-package/profiles/{profile_name}/kibana-custom.yml` ### Template Errors - Use `{{ fact "variable_name" }}` syntax for template variables @@ -168,7 +162,7 @@ You can have different custom configurations for different profiles: ~/.elastic-package/profiles/production/kibana-custom.yml ``` -Each profile can enable or disable custom configuration independently. +Custom configuration is automatically detected and applied for each profile when the `kibana-custom.yml` file exists. ## Best Practices diff --git a/internal/profile/_static/config.yml.example b/internal/profile/_static/config.yml.example index e81f99fe2e..d9f00f00a8 100644 --- a/internal/profile/_static/config.yml.example +++ b/internal/profile/_static/config.yml.example @@ -29,7 +29,3 @@ ## Set license subscription # stack.elastic_subscription: "basic" - -## Custom Kibana Configuration -# Enable custom kibana.yml configuration file -# stack.kibana_custom_config_enabled: true diff --git a/internal/stack/kibana_config.go b/internal/stack/kibana_config.go index 50b23c42d4..1f25a5276f 100644 --- a/internal/stack/kibana_config.go +++ b/internal/stack/kibana_config.go @@ -33,11 +33,7 @@ func kibanaConfigWithCustomContent(profile *profile.Profile) func(resource.Conte return fmt.Errorf("failed to write base kibana config: %w", err) } - // Check if custom config is enabled and exists - if profile.Config(configKibanaCustomConfigEnabled, "false") == "false" { - return nil // No custom config needed - } - + // Check if custom config file exists customConfigPath := profile.Path(KibanaCustomConfigFile) customConfigData, err := os.ReadFile(customConfigPath) if os.IsNotExist(err) { diff --git a/internal/stack/resources.go b/internal/stack/resources.go index 8c90c7c711..aadc7bd584 100644 --- a/internal/stack/resources.go +++ b/internal/stack/resources.go @@ -61,14 +61,13 @@ const ( elasticsearchUsername = "elastic" elasticsearchPassword = "changeme" - configAPMEnabled = "stack.apm_enabled" - configGeoIPDir = "stack.geoip_dir" - configKibanaHTTP2Enabled = "stack.kibana_http2_enabled" - configKibanaCustomConfigEnabled = "stack.kibana_custom_config_enabled" - configLogsDBEnabled = "stack.logsdb_enabled" - configLogstashEnabled = "stack.logstash_enabled" - configSelfMonitorEnabled = "stack.self_monitor_enabled" - configElasticSubscription = "stack.elastic_subscription" + configAPMEnabled = "stack.apm_enabled" + configGeoIPDir = "stack.geoip_dir" + configKibanaHTTP2Enabled = "stack.kibana_http2_enabled" + configLogsDBEnabled = "stack.logsdb_enabled" + configLogstashEnabled = "stack.logstash_enabled" + configSelfMonitorEnabled = "stack.self_monitor_enabled" + configElasticSubscription = "stack.elastic_subscription" ) var ( From 0c4b7a89c68ffda84bd3a56edbdf0a69f1fbddab Mon Sep 17 00:00:00 2001 From: Nicolas Ruflin Date: Mon, 22 Sep 2025 15:58:06 +0200 Subject: [PATCH 3/3] lint check for readme? --- README.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/README.md b/README.md index 08ba7bfd05..0dbdab90a4 100644 --- a/README.md +++ b/README.md @@ -690,13 +690,6 @@ The following settings are available per profile: Currently, it is supported "basic" and "[trial](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-trial.html)", which enables all subscription features for 30 days. Defaults to "trial". -## Custom Kibana Configuration - -You can provide custom Kibana configuration by creating a `kibana-custom.yml` file in your profile directory. -The custom configuration will be automatically appended to the base configuration when the file exists. - -See [Custom Kibana Configuration](docs/howto/custom_kibana_config.md) for detailed instructions. - ## Useful environment variables There are available some environment variables that could be used to change some of the