PipeFunc: Automate Python DAGs for Scientific Workflows - From Local to HPC with N-D Parameter Sweeps
pipefunc is a Python library that simplifies function composition and pipeline creation for scientific computing and data processing. It automatically generates DAGs from Python functions, efficiently handles N-dimensional parameter sweeps, and scales seamlessly from local to HPC environments. Key features include auto-parallization, N-dimensional map-reduce, and type checking. pipefunc minimizes boilerplate code, allowing users to focus on logic while it manages function dependencies and execution order. Ideal for complex workflows in data science, ML, and scientific computing.