Conduit shouldn't recursively traverse the pipelines directory

[hariso]

Feature description

This came up in Discord https://discord.com/channels/828680256877363200/1360147516703637666/1360591403469770902.

[rauanmayemir]

If pipelines directory is mounted from ConfigMaps in Kubernetes environment with each configmap key mapped to a file, the parsed pipeline won't be started due to ERR provisioning failed error="2 pipelines with ID \"pipeline_name\" will be skipped: duplicated pipeline ID".

The reason for that is in the way kubernetes mounts configmaps:

So when conduit traverses the directory it reads the mounted yaml file and then also resolves the hidden symlink causing duplicate error.

[lovromazgon]

Sidenote: we could reuse the code for loading connector plugins, that part doesn't traverse sub-directories.

[lyuboxa]

Sidenote: we could reuse the code for loading connector plugins, that part doesn't traverse sub-directories.

This and also when walking the dir omit any files that don't have the right extension and only load up .yml and .yaml