Authors: Maxine Hartnett, IMAP SDC
Abstract: As cloud based processing becomes more common in the scientific sphere, a huge variety of new tools and techniques are emerging for mission data pipelines. Upcoming missions are getting a new opportunity to explore these techniques from the ground-up. IMAP, a heliophysics mission launching in 2025, is able to develop an entire pipeline with a cloud-first attitude. The IMAP science data center, based out of LASP, uses tools such as AWS, infrastructure as code, and docker to create a flexible, reliable, and efficient event-based processing pipeline. The IMAP mission has 10 instruments, all of which are interdependent, which cover a broad scope of scientific data. By creating an event-based system in the cloud, the SDC can extend and modify processing based on changing requirements, while also ensuring that processing occurs quickly and reliably. This science data system takes advantage of the cloud's new ecosystem to create a pipeline that runs only what is needed, when it's needed, using small, distinct pieces of code that are easy to maintain and modify by the entire team. As the cloud becomes more wildely used, it is time to rethink the way we create processing pipelines so we can take advantage of the powerful opportunities provided by AWS and other cloud providers.
Keywords: python, aws, cloud