Optimise e2e tests performed locally against a KinD cluster

[ChristianZaccaria]

WHY

Currently, when running the e2e tests locally against a KinD cluster, the tests pass after around 30 to 40 minutes depending on the environment. Optimizing this would help speed up development and tests performed.

This issue could be due to not having enough compute resources, and some of the time is also taken from the downloading of Ray image (which is huge).

HOW

Suggestions for how this may be solved:

• Adjusting the compute resources so it runs faster.
• Caching dependencies.
• TBD

[ChristianZaccaria]

Useful Slack thread for more context.

[ChristianZaccaria]

@astefanutti Hi Antonin, I'm tagging you here in case you would like to add more info on this issue 👍

[astefanutti]

@ChristianZaccaria Thanks, that captures it very well 👍🏼.

Thinking about it more for the compute resources that are currently too limited, it could be possible to read the compute resources that are available in the cluster, and size the limits in the tests accordingly, so it dynamically adapts to the environment, and runs as fast as it could.