Refreshing external code in long-running flows

[johncalab]

I'm trying to find a solution for the following. I have a flow that is meant to run forever. A task in that flow uses an externally imported library (let's call it mylib), which may change during the lifetime of the flow. How can I make sure I'm always using the latest version of mylib?

I was thinking something along these lines, but I'm not sure it works and I'd like to know if there's a better solution for it.

from datetime import timedelta
import prefect
import mylib

install_task = prefect.tasks.shell.ShellTask(
    command="pip install mylib",
    helper_script="pip uninstall mylib",
)

@prefect.task
def task0():
    # dummy extract
    return [1,2,3]

@prefect.task
def task1(df):
    import importlib
    importlib.reload(mylib)
    df = mylib.transform(df)
    return df

with prefect.Flow() as flow:
    df_raw = task0()
    shell = install_task()
    df_transformed = task1(df_raw)
    df_transformed.set_upstream(shell)

if __name__ == '__main__':
    flow.schedule = IntervalSchedule(interval = timedelta(days=1))
    flow.run()

The point is that the pip install command will actually point to a url, so that it gets the freshest code on master every time.

[cicdw]

Once a module is imported once it won't be re-imported even if you update it in the background (as your example here does); to do this correctly, you should use a Prefect backend (Cloud would be easiest) and specify a docker image for your Flow's Run Config that you routinely update asynchronously. This way, each run of the flow will use the latest version of your Docker build without updating your Flow code.

[johncalab]

Thanks! I suspected flow would interfere with this strategy, this is unfortunate. Sadly I don't think I can use a prefect backend, as the flow runs already inside a docker container, and I don't think I have docker-in-docker support. Any other ideas on how to make this work?

[cicdw]

I don't think I follow, especially if you've already containerized the flow? If you use, for example, a Docker agent w/ a Docker run config, then you have essentially two options:

• use Docker storage with a custom dockerfile that will package your flow into a docker container; this is probably not ideal for your situation as there is a 1-1 mapping from flow <-> Docker image, and if you're updating this library regularly you might not want to update the flow every time
• use some other storage and specify an image on your Docker RunConfig (flow.run_config = DockerRun(image="example/image-name:with-tag")) -- your agent will then pull this image each time your flow runs (allowing you to update it as needed) and run the flow within the image

The fact that modules aren't reloaded is a fundamental python constraint, so you need to have some other service run this flow in a new process each time, which is exactly what the backends are for.

[johncalab]

Thanks again. Pardon my ignorance, what do you mean by Docker agent?
I think I understand the second solution you suggest, but I don't follow the first. My issue is that I do not have full control over the deployment environment. Right now what I can do is to pass a Dockerfile + code. This is then deployed in a Docker container, but that part is not under my control.
In a nutshell, what I can currently do is to write a flow.py and a Dockerfile. The Dockerfile has a command to copy flow.py and the entrypoint is python flow.py. That's why I mentioned docker-in-docker earlier (and why I also haven't found a way to use prefect's UI).
Am I making more sense now?

[johncalab]

I'm just reading now the link to Flow run Config you provided, which answered my question about Docker Agent. I don't have access to a docker daemon, at least not in my current setup.