-
Notifications
You must be signed in to change notification settings - Fork 3.5k
Doc: Add Logstash-to-serverless page #18034
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
--- | ||
mapped_pages: | ||
- https://www.elastic.co/guide/en/logstash/current/connecting-to-cloud.html | ||
--- | ||
|
||
# Sending data to {{es-serverless}} [logstash-to-elasticsearch-serverless] | ||
|
||
When you use Elasticsearch on Elastic Cloud Serverless you don’t need to worry about managing the infrastructure that keeps Elasticsearch distributed and available. These resources are automated on the serverless platform and are designed to scale up and down with your workload. | ||
|
||
::::{admonition} {{ls}} to {{serverless-full}} | ||
You’ll use the {{ls}} [{{es}} output plugin](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md) to send data to {{serverless-full}}. | ||
Note these differences between {{es-serverless}} and both {{ech}} and self-managed {{es}}: | ||
|
||
* Use [**API keys**](/reference/secure-connection.md#ls-api-keys) to access {{serverless-full}} from {{ls}} as it does not support native user authentication. | ||
Any user-based security settings in your [{{es}} output plugin](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md) configuration are ignored and may cause errors. | ||
* {{serverless-full}} uses **data streams** and [{{dlm}} ({{dlm-init}})](docs-content://manage-data/lifecycle/data-stream.md) instead of {{ilm}} ({{ilm-init}}). Any {{ilm-init}} settings in your [{{es}} output plugin](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md) configuration are ignored and may cause errors. | ||
* **{{ls}} monitoring** is available through the [{{ls}} Integration](https://github.com/elastic/integrations/blob/main/packages/logstash/_dev/build/docs/README.md) in [Elastic Observability](docs-content://solutions/observability.md) on {{serverless-full}}. | ||
|
||
**Known issue for Logstash to Elasticsearch Serverless.** | ||
The logstash-output-elasticsearch `hosts` setting defaults to port :9200. | ||
Set the value to port :443 instead. | ||
|
||
:::: | ||
|
||
## Communication between {{ls}} {{es-serverless}} [connecting-to-elasticsearch-serverless] | ||
|
||
[{{es-serverless}}](docs-content://solutions/search/serverless-elasticsearch-get-started.md) simplifies safe, secure communication between {{ls}} and {{es}}. | ||
When you configure the Elasticsearch output plugin to use [`cloud_id`](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-cloud_id) and an [`api_key`](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-api_key), no additional SSL configuration is needed. | ||
|
||
Example: | ||
|
||
* `output {elasticsearch { cloud_id => "<cloud id>" api_key => "<api key>" } }` | ||
|
||
Note that the value of the [`api_key` option](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-api_key) is in the format `id:api_key`, where `id` and `api_key` are the values returned by the [Create API key API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-api-key). | ||
|
||
|
||
### Cloud ID [cloud-id] | ||
|
||
{{ls}} uses the Cloud ID, found in the Elastic Cloud web console, to build the Elasticsearch and Kibana hosts settings. It is a base64 encoded text value of about 120 characters made up of upper and lower case letters and numbers. If you have several Cloud IDs, you can add a label, which is ignored internally, to help you tell them apart. To add a label, prefix your Cloud ID with a label and a `:` separator in this format "<label>:<cloud-id>". | ||
|
||
|
||
### API key [api-key] | ||
|
||
When you create an API key for {{ls}}, select **Logstash** from the **API key format** dropdown. | ||
This option formats the API key in the correct `id:api_key` format required by {{ls}}. | ||
|
||
:::{image} images/logstash_api_key_format.png | ||
:alt: API key format dropdown set to {{ls}}: | ||
:screenshot: | ||
:width: 400px | ||
::: | ||
|
||
The UI for API keys may look different depending on the deployment type. | ||
|
||
|
||
## Sending {{ls}} management data to {{es-serverless}} [mgmt-data] | ||
karenzone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
This setting in the `logstash.yml` config file can help you get set up to send management data to Elastic Cloud: | ||
karenzone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
* `xpack.management.elasticsearch.cloud_id` | ||
|
||
You can use the `xpack.management.elasticsearch.cloud_id` setting as an alternative to `xpack.management.elasticsearch.hosts`. | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we also need to document the Elasticsearch input and Elasticsearch filter here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This suggestion warrants discussion. The content in this PR is currently unavailable in our docs, and I don't want to hold it up. Please elaborate on what you think we need to say about input and filter in this context, and we can handle in a followup PR. I'll open a new issue if we need it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My thinking here is that everything that we state around cloud IDs, and API keys in the
Communication between LS and ES-serverless
section that mention the Elasticsearch output, also hold for the Elasticsearch input and filter plugins.