Merge pull request #1 from pyronear/rs/add-adr

ADD ADR (Architectural Decision Record) about workflow management system

ADR/Luigi

@@ -0,0 +1,49 @@
+# Title: Workflow Management for Fire Detection Pipeline
+
+Status: Luigi rejected, Python script selected
+Deciders: Ronan Sy
+Date: October 2024
+
+# Context and Problem Statement
+
+We needed a workflow management tool to automate the process of downloading images and running fire detection algorithms on them. The process involves several sequential tasks: downloading images, analyzing them for fires, and, if a fire is detected, storing the detection in the database. However, the pipeline is simple, and we rarely detect fires, which makes the management of this process less critical. The goal was to find a tool that could manage this workflow efficiently, without excessive overhead.
+
+# Decision Drivers
+
+- Simplicity: Our pipeline is simple and doesn’t require complex orchestration.
+- Maintainability: We want to avoid introducing unnecessary complexity into our codebase.
+- Criticality: The last step of recording fire detections is critical, but it is rarely triggered.
+- Error handling: Some failures, such as image download issues, are not critical, and we can afford some degree of failure in these areas.
+
+# Considered Options
+
+Tools like Airflow and Dagster were not considered since there docker architecture would add too much overhead to the current pipleline. We considered one main option compared to our current Python script : Luigi 
+
+Pros:
+- Direct integration of parameters and dependency management between tasks  into Python code.
+- "Target" concept ensures tasks aren’t re-executed if their output already exists. More generally, good practices are enforced by Luigi since you need to obey to the task structure. 
+- command-line interface allows executing specific tasks.
+
+Cons:
+- Limited scalability.
+- Reusability of tasks is not straightforward.
+- Poor UI, no easy manual triggering or interaction.
+- Unnecessary overhead in the code for our simple use case.
+
+# Decision Outcome
+
+We decided to reject Luigi for now and proceed with a single Python script to manage the pipeline. Luigi introduces complexity that isn’t justified given the simplicity and non-criticality of our pipeline, except for the rare case of fire detection. Instead, we will enhance error handling and logging in our Python code to ensure robustness, especially for the final critical step.
+
+# Pros and Cons of the Decision
+
+Pros:
+- Simple, easy-to-maintain solution without the overhead of a workflow management tool.
+- Can focus development effort on improving error handling and ensuring reliability in the final detection step.
+
+Cons:
+- No automated task dependency management or tracking.
+- Limited scalability if the pipeline evolves into something more complex in the future.
+
+# Mitigation Strategy
+
+If the need arises for more complex workflows in the future, we can revisit Luigi or another orchestration tool. For now, we will focus on improving logging and error handling in the Python script to ensure the pipeline runs smoothly, particularly for the critical fire detection recording step.