Implement Kibana/OpenSearch Dashboards dev services

[Javatar81]

This change will add dashboard services to the dev services for extensions/elasticsearch-rest-client and extensions/elasticsearch-java-client. If the distribution is elasticsearch, it will start Kibana as the dashboard. If the distribution is opensearch, it will start OpenSearch-dashboards as the dashboard.

• Fixes Kibana/OpenSearch Dashboards dev services #39582

[quarkus-bot]

/cc @gsmet (elasticsearch), @loicmathieu (elasticsearch), @marko-bekhta (elasticsearch)

[andreaTP]

cc. @yrodiere @holly-cummins

[marko-bekhta]

Hey 👋🏻

thanks for the PR! 🙂

Could you please use rebase instead of adding merge commits? Something like:

git checkout main
git fetch upstream
git rebase upstream/main

where upstream points to this repo (git remote add upstream https://github.com/quarkusio/quarkus.git that is if you haven't set up the upstream yet) usually should do it for your current branch (main from which you've opened this PR) 🙂

I think it would be better if you just create another processor for the dashboards (i.e. DevServicesElasticsearchDashboardsProcessor), that way you'd easiely spot if you are not missing any stop/restart actions + the processor for the Elasticsearch services is about to change to work with multiple services at the same time.

You've made good progress here already 👍🏻
Have you had a chance to look into this part about dev ui as well:

it would be a great start if we provided either an option to launch Kibana in the Elasticsearch dev services (and include a link to it in the Dev UI),

it can go as a follow up if not 🙂

[Javatar81]

Thanks for all your comments. So I would move all my changes into a separate class DevServicesDashboardProcessor with its own properties files and address your proposed changes.

Regarding the Dev Ui. Yes I had that in mind but I planned it as a follow up.

[gsmet]

Thanks for this work, I see we are making good progress. I added a small question.

[Javatar81]

I am in the refactoring to move everything related to dashboards into a new processor class named DevServicesElasticsearchDashboardsProcessor. Since both are running in parallel I must make sure that the elasticsearch dev service is already running (but only if the hosts variable is not set). One obvious solution would be to repeat locating the elasticsearch container with a backoff. Is there a more elegant way to wait for the other devservice to be started?

[marko-bekhta]

I am in the refactoring to move everything related to dashboards into a new processor class named DevServicesElasticsearchDashboardsProcessor. Since both are running in parallel I must make sure that the elasticsearch dev service is already running (but only if the hosts variable is not set). One obvious solution would be to repeat locating the elasticsearch container with a backoff. Is there a more elegant way to wait for the other devservice to be started?

oh yes, you could make the ES services produce a build item(s) for you (it would contain a list of hosts) and then you'd use that build items as an input to your dashboards step:

...
startElasticsearchDevService(
    ...
    BuildProducer<ElasticsearchHostBuildItem> elasticsearchHosts
    .. .) {
        ... 

        elasticsearchHosts.produce(...)

    }

and then in the dashboards:

startElasticsearchDashboardsDevService(
    ...
    List<ElasticsearchHostBuildItem> elasticsearchHosts
    .. .) {
        ...
        List<String> eshosts = getAllEsHosts(elasticsearchHosts);
        ..
    }

(maybe you could come up with a better name for ElasticsearchHostBuildItem as well 🙂)

[Javatar81]

All points addressed so far. Please review again.

[github-actions]

🎊 PR Preview f8ca654 has been successfully built and deployed to https://quarkus-pr-main-51164-preview.surge.sh/version/main/guides/

• Images of blog posts older than 3 months are not available.
• Newsletters older than 3 months are not available.

[Javatar81]

If anything else is needed, please let me know.

[marko-bekhta]

hey 👋🏻 🙂

sorry for not getting back to you sooner. Thanks for splitting out the Kibana processor!
I've added a couple comments where we could cleanup a few things.

[marko-bekhta]

Suggested change

Optional<ContainerAddress> maybeContainerAddressSearchBackend = Optional.empty();

seems to be unused ...

[marko-bekhta]

Thinking more about this ... hosts are a runtime property:

quarkus/extensions/elasticsearch-rest-client/runtime/src/main/java/io/quarkus/elasticsearch/restclient/lowlevel/runtime/ElasticsearchConfig.java

Line 22 in 15c3f56

List<InetSocketAddress> hosts();

so we won't necessarily get them here...
Do you happen to know if there's an easy way we can "append" the list of hosts we pass to kibana/dashboards once they've starteed and are running ? (maybe some REST API ?)

[marko-bekhta]

Thinking more about it ...
maybe let's start with using just List<DevservicesElasticsearchConnectionBuildItem> elasticsearchConnectionBuildItems for now. And find a way to add the non-devservices instances later.

[Javatar81]

If quarkus.elasticsearch.hosts is set, the Elasticsearch dev service is not started, and we don't have anyDevservicesElasticsearchConnectionBuildItems. In this case the Dashboard processor reads the quarkus.elasticsearch.hosts.

[marko-bekhta]

Suggested change

	log.info(
	"no elasticsearch hosts config property found, using the host of theelasticsearch dev services container to connect");
	log.debug(
	"No Elasticsearch hosts config property found, using the host of the Elasticsearch dev services container to connect");

[marko-bekhta]

Suggested change

LaunchModeBuildItem launchMode, DevServicesComposeProjectBuildItem composeProjectBuildItem,

I think we only needed these to lookup the container, but you've done that already before calling these create methods, so maybe let's remove the paramteres here ?

[marko-bekhta]

If we go with the idea of using just the DevservicesElasticsearchConnectionBuildItem and having the distribution in there, we probably can get most of these methods back into this processor and keep them private.

[Javatar81]

Not this one because if quarkus.elasticsearch.hosts is set, the Elasticsearch dev service is not started, and we don't have anyDevservicesElasticsearchConnectionBuildItems. In this case the Dashboard processor reads the quarkus.elasticsearch.hosts and can only get the distribution from quarkus.elasticsearch.devservices.distribution.