Support multi node integ tests

[owaiskazi19]

Description

Support multi node integ tests.
Improvements done for CI in this PR:

1. example 1, example 2 integ test was running twice in precommit and check. Removed running integTest again in precommit
2. jacocotestReport tasks depends on test and integTest. Since gradle check already runs both, moved codeCov CI under gradle check.
3. Added multi node integration tests CI

Related Issues

Resolves ##1307

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[vibrantvarun]

Gradle check is failing.

[yuye-aws]

@owaiskazi19 Can you update the change log?

[owaiskazi19]

Unrelated test is failing on main as well

REPRODUCE WITH: ./gradlew ':test' --tests "org.opensearch.neuralsearch.search.query.HybridCollectorManagerTests.testRescoreWithConcurrentSegmentSearch_whenMatchedDocsAndRescore_thenSuccessful" -Dtests.seed=31B75FEED86AF287 -Dtests.security.manager=false -Dtests.locale=fi-FI -Dtests.timezone=Europe/Brussels -Druntime.java=23

HybridCollectorManagerTests > testRescoreWithConcurrentSegmentSearch_whenMatchedDocsAndRescore_thenSuccessful FAILED
    java.lang.AssertionError: expected:<3> but was:<1>
        at __randomizedtesting.SeedInfo.seed([31B75FEED86AF287:BB4DF1B8A1D4DC19]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.neuralsearch.search.query.HybridCollectorManagerTests.testRescoreWithConcurrentSegmentSearch_whenMatchedDocsAndRescore_thenSuccessful(HybridCollectorManagerTests.java:1019)

[yuye-aws]

Looks good to me! Thanks for the PR @owaiskazi19 !

[yuye-aws]

Not have much context on the current flakey tests. Can @vibrantvarun help verify?

[junqiu-lei]

LGTM, thank you! @owaiskazi19

[heemin32]

Thanks for the PR. I have one more request—could you set the default number of shards to 3 for the entire test? This will ensure the test properly exercises the multi-node scenario.

I'm not sure if you mean to move or copy the entire test suite for a multi-node setup. I think we need a one node tests, we had multiple cases in the past where issue appears only in a single shard/node setup. I suggest we have both single and multi node setup as part of CI

I thought we are sharing test cases between single node and multi node? I that case, setting shard number to be 3 even for single node won't have any harm?

[martin-gaievski]

Thanks for the PR. I have one more request—could you set the default number of shards to 3 for the entire test? This will ensure the test properly exercises the multi-node scenario.

I'm not sure if you mean to move or copy the entire test suite for a multi-node setup. I think we need a one node tests, we had multiple cases in the past where issue appears only in a single shard/node setup. I suggest we have both single and multi node setup as part of CI

I thought we are sharing test cases between single node and multi node? I that case, setting shard number to be 3 even for single node won't have any harm?

We can only change number of nodes from the outside of test logic.

If we have whole test suite running on both single node and multiple nodes that's fine, we have more coverage. Where I do have concerns is if we remove tests that run on a single node and will run everything on a multiple nodes. At first glance this should lead to a yellow cluster state if we have only one shard without replicas.

[heemin32]

At first glance this should lead to a yellow cluster state if we have only one shard without replicas.

This is true for single node as well. Cluster health will be yellow if there is no replica even with single node.

[martin-gaievski]

At first glance this should lead to a yellow cluster state if we have only one shard without replicas.

This is true for single node as well. Cluster health will be yellow if there is no replica even with single node.

Yes, you're correct, this will be same for a single node cluster, then it should be fine to run all tests on a multi node cluster