Docker tasks don't seem to be pulling the right image

[dusktreader]

I'm sure I'm not doing something right here, but I've hit a spot where I have my Prefect infrastructure all up and flows registered. However, when the tasks kick off, they seem to be trying to pull the wrong images from the wrong repository.

So, I have a running Docker Agent which is connected and visible in the ui. When I run a flow with docker tasks, however, it does not seem to do the right things. My flow code is:

import prefect

create_task = prefect.tasks.docker.CreateContainer()
start_task = prefect.tasks.docker.StartContainer()
status_task = prefect.tasks.docker.WaitOnContainer()


with prefect.Flow("hello-flow") as flow:
    container_id = create_task(
        image_name="python:3.9-slim",
        command='python -c "print "Hello Universe!"',
    )
    start_task(container_id=container_id)
    exit_code = status_task(container_id=container_id)
flow.register(project_name="test-project")

When I schedule the flow, though, the flow fails immediately:

I've tried a bunch of different approaches, but I feel like I'm missing something when it comes to running tasks with docker agents. I can share additional source code or configuration as needed to illustrate the problem. Any help would be very appreciated!

[dusktreader]

I just found this thread, and it was pretty illuminating: #2361

Since I'm starting my agent from within a docker container, I needed to pass the docker_interface=False flag in my call to DockerAgent().

Additionally, it seems like I need to add a DockerRun config to my flow.

I have a new issue now (404 Client Error for http+docker://localhost/v1.39/containers/create: Not Found ("No such image: python:3.9-slim")...

But, at least this is progress

[joshmeek]

Perhaps you need to call the PullImage task first to make sure it exists before you attempt to create the container

[dusktreader]

I had tried that before setting docker_interface=False, so it warrants a new try.

However, the logs seem to show that the flow tasks aren't even being started. It's just trying to start the container for the flow and failing there. The latest error after my tinkering is: ERROR - agent | Error while deploying flow: ImageNotFound(HTTPError('404 Client Error: Not Found for url: http+docker://localhost/v1.39/containers/create'))

It seems like the agent itself doesn't have docker available inside its container...not sure how to address that.

[dusktreader]

Source code here: https://github.com/dusktreader/swarm-prefect

[dusktreader]

Well, I've made a lot of progress on this, but now I'm hitting a new befuddling issue:

trillian_1       | [2020-12-31 20:51:26+0000] ERROR - prefect.Local | Failed to load Flow from /root/.prefect/flows/hello-flow.prefect
trillian_1       | Traceback (most recent call last):
trillian_1       |   File "/usr/local/lib/python3.7/site-packages/prefect/storage/local.py", line 105, in get_flow
trillian_1       |     return extract_flow_from_module(module_str=flow_location)
trillian_1       |   File "/usr/local/lib/python3.7/site-packages/prefect/utilities/storage.py", line 123, in extract_flow_from_module
trillian_1       |     module = importlib.import_module(module_name)
trillian_1       |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
trillian_1       |     return _bootstrap._gcd_import(name[level:], package, level)
trillian_1       |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
trillian_1       |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
trillian_1       |   File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
trillian_1       |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
trillian_1       |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
trillian_1       |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
trillian_1       |   File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
trillian_1       |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
trillian_1       |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
trillian_1       |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
trillian_1       |   File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
trillian_1       | ModuleNotFoundError: No module named '/root/'
trillian_1       | No module named '/root/'

Do I need to volume mount my flows directory into the the DockerAgent at /root/.prefect?

I don't see anything about this in the docs

[joshmeek]

What storage option are you using? Looking at the repo linked above the storage seems to be commented out which means it will default to Local storage and the Docker agent won't be able to run the flow if it is stored in your local filesystem 🤔