Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Starlette server fails with top level async-await #1964

Open
C-Loftus opened this issue Mar 12, 2025 · 2 comments
Open

Starlette server fails with top level async-await #1964

C-Loftus opened this issue Mar 12, 2025 · 2 comments
Labels
bug Something isn't working

Comments

@C-Loftus
Copy link
Contributor

C-Loftus commented Mar 12, 2025

Description

When launching pygeoapi with pygeoapi serve --starlette pygeoapi throws an error if you try to use top level async functions.
I expected to be able to use async functions without needing to use asyncio.run since the web server should initialize the async event loop.

If you are developing a large pygeoapi plugin with lots of fetch-based ETL, async is often needed and using asyncio.run over every async function adds a fair bit of boilerplate (and requires you to make sure another event loop isn't running / created by another plugin)

The error seems to signify that the content is not being awaited properly

Steps to Reproduce

  1. Create a custom provider plugin
  2. Create an async function like get inside the provider
    async def get(self, identifier, **kwargs):
        await asyncio.sleep(1)
    # rest of geojson response omitted here for brevity ...
  1. When you call the oaf endpoint and thus call get() it will give the error:
Traceback (most recent call last):
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 460, in handle
    await self.app(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/pygeoapi/starlette_app.py", line 360, in collection_items
    return await execute_from_starlette(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/pygeoapi/starlette_app.py", line 133, in execute_from_starlette
    headers, status, content = await loop.run_in_executor(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cloftus/.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/pygeoapi/starlette_app.py", line 104, in call_api_threadsafe
    return api_call(*args)
           ^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/pygeoapi/api/itemtypes.py", line 856, in get_collection_item
    if 'links' not in content:
       ^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'coroutine' is not iterable

Expected behavior

I expect to be able to have starlette itself initialize the async event loop. I expected to be able to use async functions as it mentions in the starlette docs

from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route


async def homepage(request):
    return JSONResponse({'hello': 'world'})


app = Starlette(debug=True, routes=[
    Route('/', homepage),
])

Environment

  • OS: Macos Sonoma 14.3
  • Python version: 3.12.8
  • pygeoapi version: pygeoapi, version 0.20.dev0
@C-Loftus C-Loftus added the bug Something isn't working label Mar 12, 2025
@C-Loftus
Copy link
Contributor Author

C-Loftus commented Mar 12, 2025

A similar issue is generated if you try to use the EDR provider instead of the OAF provider for a plugin. It appears in both cases a coroutine is not being awaited properly

    @BaseEDRProvider.register()
    async def area(
        self,
        wkt: str,
        select_properties: list[str] = [],
        datetime_: Optional[str] = None,
        z: Optional[str] = None,
        format_: Optional[str] = None,
        **kwargs,
    ):
        await asyncio.sleep(1)
        # .... rest of valid covjson response omitted here for brevity; but it doesn't get beyond the sleep regardless
Traceback (most recent call last):
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 460, in handle
    await self.app(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/pygeoapi/starlette_app.py", line 506, in get_collection_edr_query
    return await execute_from_starlette(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/pygeoapi/starlette_app.py", line 133, in execute_from_starlette
    headers, status, content = await loop.run_in_executor(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cloftus/.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/pygeoapi/starlette_app.py", line 104, in call_api_threadsafe
    return api_call(*args)
           ^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/pygeoapi/api/environmental_data_retrieval.py", line 420, in get_collection_edr_query
    content = to_json(data, api.pretty_print)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/pygeoapi/util.py", line 284, in to_json
    return json.dumps(dict_, default=json_serial, indent=indent,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cloftus/.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
          ^^^^^^^^^^^
  File "/Users/cloftus/.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/json/encoder.py", line 202, in encode
    chunks = list(chunks)
             ^^^^^^^^^^^^
  File "/Users/cloftus/.local/share/uv/python/cpython-3.12.8-macos-aarch64-none/lib/python3.12/json/encoder.py", line 439, in _iterencode
    o = _default(o)
        ^^^^^^^^^^^
  File "/Users/cloftus/github/RISE-EDR/.venv/lib/python3.12/site-packages/pygeoapi/util.py", line 417, in json_serial
    raise TypeError(msg)
TypeError: <coroutine object RiseEDRProvider.area at 0x10f5d1f30> type <class 'coroutine'> not serializable

@webb-ben
Copy link
Member

@francbartoli @ricardogsilva interested to hear you experience developing async within pygeoapi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants