-
Notifications
You must be signed in to change notification settings - Fork 96
Open
Labels
bugSomething isn't workingSomething isn't working
Description
I'm using an async generator function to produce a list of jobs - currently reading from a file but eventually it will be a database query.
Right now, when I pass the generator into from_iterable directly, my downstream map functions show type errors
async def queue_jobs() -> typing.AsyncGenerator[str, None]:
with open(
os.path.join(Path(__file__).parent, 'urls.txt')
) as f:
for url in f.read().splitlines():
yield url.strip()
data = pl.task.from_iterable(queue_jobs(), maxsize=10)
data = pl.task.map(fetch_url, stage=data, workers=10)
data = pl.task.map(process_response, stage=data, workers=10)
Both fetch_url and process_response have the error:
Type "(url: str) -> Coroutine[Any, Any, FetchResponse]" cannot be assigned to type "(A: Unknown, **kwargs: Unknown) -> (B@__call__ | Awaitable[B@__call__])"
Parameter name mismatch: "A" versus "url"
Parameter "**kwargs" has no corresponding ```
Both fetch_url and process_response are async functions:
async def fetch_url(url: str) -> FetchResponse:
....
async def process_response(response: FetchResponse) -> Article:
....
Where `FetchResponse` and `Article` are both Pydantic classes.
Using Python 3.11.3, Pypeln 0.4.9
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working