feature request: support pyc files for flow deployments.

[marchmelo0923]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

When Flow or Task function is inside .pyc (compiled module),
It fails on run deployment, while serve is success

Reproduction

Sorry, but I can not give my code because of a security issueThe following is an example that probably causes the same problem.

ml_pipeline
  - __init__.pyc
  - train_flow.pyc
main.pyc

[main.py]
...
def start_server(port):
    p = Process(target = _start_server_worker, args=(port,))
    p.start()

def _start_server_worker(port):
    app(["server", "start", "--port", port])

def load_train_process(distributed):
    p = Process(target = _load_train_process_worker, args=(distributed,))
    p.start()

def _load_train_process_worker(distributed):
    import ml_pipeline.train_flow
    classification_model_flow = ml_pipeline.train_flow.ClassificationModelFlow
    classification_deploy = ml_pipeline.train_flow.ClassificationModelFlow.to_deployment()
    serve(classification_deploy, print_starting_message=False)
..

my train_flow.py has prefect flow function... but I can't give it. Sorry

Error

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "C:\code\(Hide Addresses)\python-3.11.9-embed-amd64\Lib\site-packages\prefect\utilities\importtools.py", line 187, in load_module
    return importlib.import_module(module_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "importlib\__init__.py", line 126, in import_module
  File "", line 1204, in _gcd_import
  File "", line 1176, in _find_and_load
  File "", line 1126, in _find_and_load_unlocked
  File "", line 241, in _call_with_frames_removed
  File "", line 1204, in _gcd_import
  File "", line 1176, in _find_and_load
  File "", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'ml_pipeline\\train_flow'

Versions (prefect version output)

prefect==2.19.6

Additional context

I think it maybe importlib's problem, so I edit prefect code like this

try:
    return importlib.import_module(module_name)
finally:
    sys.path.remove(working_directory)

to

try:
    return importlib.import_module(module_name)
except:
    spec = importlib.util.spec_from_file_location(module_name)
    module = importlib.util.module_from_spec(spec)
    return module
finally:
    sys.path.remove(working_directory)

In this case, it got error like this....

Traceback (most recent call last):
File "C:\code(Hide Addresses)\python-3.11.9-embed-amd64\Lib\site-packages\prefect\engine.py", line 422, in retrieve_flow_then_begin_flow_run
else await load_flow_from_flow_run(flow_run, client=client)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\code(Hide Addresses)\python-3.11.9-embed-amd64\Lib\site-packages\prefect\client\utilities.py", line 100, in with_injected_client
return await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\code(Hide Addresses)\python-3.11.9-embed-amd64\Lib\site-packages\prefect\deployments\deployments.py", line 317, in load_flow_from_flow_run
flow = await run_sync_in_worker_thread(load_flow_from_entrypoint, str(import_path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\code(Hide Addresses)\python-3.11.9-embed-amd64\Lib\site-packages\prefect\utilities\asyncutils.py", line 136, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\code(Hide Addresses)\python-3.11.9-embed-amd64\Lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\code(Hide Addresses)\python-3.11.9-embed-amd64\Lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "C:\code(Hide Addresses)\python-3.11.9-embed-amd64\Lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\code(Hide Addresses)\python-3.11.9-embed-amd64\Lib\site-packages\prefect\flows.py", line 1677, in load_flow_from_entrypoint
raise MissingFlowError(
prefect.exceptions.MissingFlowError: Flow function with name 'ClassificationModelFlow' not found in 'ml_pipeline\train_flow.pyc'.

[marchmelo0923]

serve deployment is fine, and the entry point is recognized normally as follows
I think there is no reason that prefect can not read the .pyc module.

[WillRaphaelson]

Thanks for the issue @marchmelo0923. An MRE would help here, even if the ClassificationModelFlow code is just substituted for pass. Let me know if you want to connect on slack to discuss in more depth where we can look at code. I dont at face value see why this pyc files would be user and/or get in the way.

[marchmelo0923]

[train_flow.py]
from prefect import flow, task

@task
def mytask1():
    pass

@task
def mytask2():
    pass

...

@task
def mytaskn():
    pass

@flow()
def ClassificationModelFlow(arg1:str, arg2:str):
    mytask1()
    mytask2()
   ...
    mytaskn()

I can't release the code, so it is just abbreviated. Structure itselt is the same as this.

And then, all .py codes in my programs are compiled by
'python -m compileall -b .'
After that, I remove orginal .py file.
When run it, It successfully open ui server, and serve deployment.

It successfully open UI server and serve deployment. So, I think It should be work fine.

[marchmelo0923]

Thanks for the issue @marchmelo0923. An MRE would help here, even if the ClassificationModelFlow code is just substituted for pass. Let me know if you want to connect on slack to discuss in more depth where we can look at code. I dont at face value see why this pyc files would be user and/or get in the way.

Can you invite me to Slack? Since I live in a place with a time difference of 13 hours with you, I don't think it's going to be easy to continue the story by asking on Github like this.

[WillRaphaelson]

The reason it works when you do .serve is that you are passing the function directly to the serve call, which we can import from the pyc file. When we deploy, we dont actually run the imports, we parse the python code to determine the flow entrypoint. Because there is no python file, we're not able to determine that entrypoint and this will not work. We can change this issue to a feature request to support deploying from pyc files to continue to gather user input, but for the time being you'll want to use the py files directly in deployments.

More information on the slack community here.

[marchmelo0923]

Um... There's nothing I can do then
In fact, the reason I tried to make it into a .pyc file was to interfere with the exposure of the code (apart from being easy to decrypt, I only need to prevent the direct exposure of the code)
So, I tried using pyinstaller in a Windows environment, but pyinstaller also didn't work well due to a package dependency collision
By any chance, other than how to make it into a .pyc file, is there a way to prevent direct code exposure supported by prefect?
I know it's a lot off the mark from the initial question, but I need some help.

[WillRaphaelson]

Depending on your infrastructure setup, you could consider dockerizing your code and dependencies and running on a docker work pool or any of the other work pools that run containers (K8s, ECS, Cloud run, ACI).

[marchmelo0923]

Thanks for your help..!!
My question is end, but i will not close this issue.. until this feature request ended