Deploy monitoring stack manually: grafana import is broken

[YuryHrytsuk]

Problem description

Since grafana state is managed via terraform, cd services/monitoring && make up-<target> does not work since there is no terraform cli tool installed on the docker swarm machines

make grafana-import
make[1]: Entering directory '...'
...
make[2]: Entering directory .../services/monitoring/grafana'
# generate terraform/main.tf
# terraform init
/bin/bash: line 5: terraform: command not found

Related Issue(s)

• Consider using grafana terraform provider instead of hacky-DYI-REST-provisioning #778

Related PR(s)

• ✨ Add: Manage grafana content/assets via terraform #913

[mrnicegyu11]

Hey @YuryHrytsuk thanks for highlighting this. Since we would not want to install too may extra packages on our deployments, and since the terraform can always be manage from your local machine (no need to be on the deployments' servers actually), I think this is a wontfix. I dont see when one would need to execute make import-grafana on the machines, can you maybe clarify? thx!

[YuryHrytsuk]

Hey @YuryHrytsuk thanks for highlighting this. Since we would not want to install too may extra packages on our deployments, and since the terraform can always be manage from your local machine (no need to be on the deployments' servers actually), I think this is a wontfix. I dont see when one would need to execute make import-grafana on the machines, can you maybe clarify? thx!

Every now and again we do need to deploy our services manually. It can be necessary, for example, to quickly fix bug on PROD or for developing purposes on Master. I presume then that this can be achieved by executing commands on your local machine, although this breaks existing convention that make up-<target> is sufficient and makes deploying less generalized and now requires specific knowledge of this particular usecase. (Let me know if there are other exceptions in our stacks).

How does it actually work in CI? deploy_ops / force_deploy_monitoring executes make up-<target> on the docker swarm node to deploy a swarm service. Does it work in CI actually?

[YuryHrytsuk]

Hey @YuryHrytsuk thanks for highlighting this. Since we would not want to install too may extra packages on our deployments, and since the terraform can always be manage from your local machine (no need to be on the deployments' servers actually), I think this is a wontfix. I dont see when one would need to execute make import-grafana on the machines, can you maybe clarify? thx!

Every now and again we do need to deploy our services manually. It can be necessary, for example, to quickly fix bug on PROD or for developing purposes on Master. I presume then that this can be achieved by executing commands on your local machine, although this breaks existing convention that make up-<target> is sufficient and makes deploying less generalized and now requires specific knowledge of this particular usecase. (Let me know if there are other exceptions in our stacks).

How does it actually work in CI? deploy_ops / force_deploy_monitoring executes make up-<target> on the docker swarm node to deploy a swarm service. Does it work in CI actually?

As I think of it further, cd services/monitoring && make up-<target> is logically broken since it is supposed to be executed on the swarm node but it calls make grafana-import which is not supposed to be run on the swarm node 🤔

[mrnicegyu11]

mhm those are good point, Im will have to dig into this a bit. But very valid, I take back the wontfix label for now :D

[mrnicegyu11]

OK, the reasonable solution is to install tfenv on all manager machines. The rest will then be taken care of automatically. @bisgaard-itis issue with the dashboards not being CD-deployed is also boiling down to this issue of terraform not being available on the deployments' swarm managers....

i'll prepare an ansible PR

[mrnicegyu11]

https://git.speag.com/oSparc/osparc-infra/-/merge_requests/295#934d52d836cf86e55f494c028edd947e7056fee7