Skip to content

[Bug] Unexpected typing behavior #104

@chrisgoddard

Description

@chrisgoddard

I'm using an async generator function to produce a list of jobs - currently reading from a file but eventually it will be a database query.

Right now, when I pass the generator into from_iterable directly, my downstream map functions show type errors


async def queue_jobs() -> typing.AsyncGenerator[str, None]:
    with open(
        os.path.join(Path(__file__).parent, 'urls.txt')
    ) as f:
      for url in f.read().splitlines():
          yield url.strip()

data = pl.task.from_iterable(queue_jobs(), maxsize=10)

data = pl.task.map(fetch_url, stage=data, workers=10)

data = pl.task.map(process_response, stage=data, workers=10)

Both fetch_url and process_response have the error:

  Type "(url: str) -> Coroutine[Any, Any, FetchResponse]" cannot be assigned to type "(A: Unknown, **kwargs: Unknown) -> (B@__call__ | Awaitable[B@__call__])"
    Parameter name mismatch: "A" versus "url"
    Parameter "**kwargs" has no corresponding ```

Both fetch_url and process_response are async functions:

async def fetch_url(url: str) -> FetchResponse:
....

async def process_response(response: FetchResponse) -> Article:
....


Where `FetchResponse` and `Article` are both Pydantic classes.


Using Python 3.11.3, Pypeln 0.4.9

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions