bug: Unable to see more than 200 logs in Generator Task Logs

[BeArchiTek]

Component

Frontend UI

Infrahub version

1.1.7

Current Behavior

When running a Generator, you can create logs and see them in the UI using the logger function.
If there is more than 200 logs, a part of it will be in another "page", but there is no options to see those.

Expected Behavior

I am able to navigate and see all the logs

Steps to Reproduce

Use the demo schema
Update any of the generator of demo-edge to add the log loop below in the generate function

        log = logging.getLogger("infrahub.tasks")

        for i in range(205):
            log.info(f"Demo log entry {i + 1}")

Additional Information

No response

[dgarros]

Would be good to check if the logs are present in Prefect in or not ?
Most-likely we need to implement some pagination when we retrieve the logs

[ogenstad]

Assuming this is the InfrahubTask query the backend should already do this. Isn't this a frontend issue?

[wvandeun]

The prefect flow show the proper log entries:

The InfrahubTask graphql query only returns 200 TaskLog nodes

[ogenstad]

Reposting info from Slack. The cut at 200 comes from an environment variable within Prefect.

                "default_limit": {
                    "default": 200,
                    "description": "The default limit applied to queries that can return multiple objects, such as `POST /flow_runs/filter`.",
                    "supported_environment_variables": [
                        "PREFECT_SERVER_API_DEFAULT_LIMIT",
                        "PREFECT_API_DEFAULT_LIMIT"
                    ],
                    "title": "Default Limit",
                    "type": "integer"
                },

@router.post("/filter", response_class=ORJSONResponse)
async def read_flow_runs(
    sort: schemas.sorting.FlowRunSort = Body(schemas.sorting.FlowRunSort.ID_DESC),
    limit: int = dependencies.LimitBody(),
    offset: int = Body(0, ge=0),
    flows: Optional[schemas.filters.FlowFilter] = None,
    flow_runs: Optional[schemas.filters.FlowRunFilter] = None,
    task_runs: Optional[schemas.filters.TaskRunFilter] = None,
    deployments: Optional[schemas.filters.DeploymentFilter] = None,
    work_pools: Optional[schemas.filters.WorkPoolFilter] = None,
    work_pool_queues: Optional[schemas.filters.WorkQueueFilter] = None,
    db: PrefectDBInterface = Depends(provide_database_interface),
) -> List[schemas.responses.FlowRunResponse]:

While we can consider some form of warning for that I wonder if this initially should be resolved within the frontend by using pagination.

[dgarros]

While we can consider some form of warning for that I wonder if this initially should be resolved within the frontend by using pagination.

Yes that would be ideal but I don't think the backend is exposing the right API for it right now

[ogenstad]

Yes that would be ideal but I don't think the backend is exposing the right API for it right now

Isn't it just this we're talking about?

query MyQuery {
  InfrahubTask(limit: 1, offset: 2) {
    count
    edges {
      node {
        title
      }
    }
  }
}

[dgarros]

No, I think the issue is that we need pagination for the logs as part of a task, not at the task level

[ogenstad]

Ah right I missed the indentation in the screenshot. Looks like Prefect doesn't support a count on that API, so we'd either have to add one ourselves in the wrapper alternatively have a loop that continues querying until no additional requests are returned when adding an offset.