Exit with error when an experiment is specified

[nathanperkins]

Description

Given that experiments can cause major changes in task behavior (like env precedence, it'd be nice to see a strict option (or maybe default behavior) where specifying an experiment that is unsupported or removed results in task exiting immediately with error.

This would avoid a dangerous situation where we get different, dangerous behavior based on what version of taskfile is running the task.

[vmaerten]

I am not sure if we want to develop something for this because Experiments were not designed to be used in production and we do not want to develop large feature for this.
However, we could imagine make experiment available in the templating system, so you could use it in preconditions

[pd93]

So this was discussed a bit further in #1982 (comment) and the subsequent comments. Thinking about it further, if a user has (for example), the Remote Taskfile experiments enabled and we then made the experiment GA, do we want to suddenly start erroring, even though their Taskfile is valid just because they have an old flag enabled?

Maybe we can print a warning saying the flag is no longer required, but erroring a Taskfile that previously worked seems annoying and would force the user to go fix the problem immediately or roll back the Task version.

To be clear, experiments like env precedence are breaking changes and cannot be made GA until a major version is cut - in which case the user will be expected to make some changes anyway.

[nathanperkins]

If changing from an experiment to native support or removing the experiment requires a major version change, a warning sounds reasonable to me :)

Mostly I'd want to avoid surprises with behavior in our CI from simply using a different minor / patch version.

Edit: although it's likely we would check the logs carefully after any version change so we're probably fine with a warning anyway :)

Edit2: the thing I'd be more concerned about (on the env var exp) is a team member using an old version which isn't aware and doesn't support the option or the GA version.

[pd93]

the thing I'd be more concerned about (on the env var exp) is a team member using an old version which isn't aware and doesn't support the option or the GA version.

Just to ease your concerns - We will bump the schema version to 4 when we bump a major version, so you won't be able to run a v4 Taskfile using v3 of Task and vice versa anyway. In other words (IMO anyway), a v3 schema's behaviour should never change unless an experiment is enabled.