Use of set_hook in docker container

[crelocks]

How to use set_hook if I am running crawl4ai in a docker?

[unclecode]

Such a good question, While typing, I think at the same time :D:D. I'm considering adding a redirect webhook that you can pass when making a call. Then the Docker server will make a synchronous call to that URL, sharing some data and notifying you. I'll think about it for a while and give it some time. Thx for pointing out.

[EcksDy]

@unclecode
I've taken a deeper look at the release notes and the commits, and I might be missing something but it might not be exactly what @berengamble and I were looking for.

We've been looking for classic webhooks - where you can provide a webhooks_url along with the crawl request, and the docker server would ping it once the crawl job is done.
Usually webhooks would either send some metadata: job id, status, URL where further details(crawl data in our case) can be found, or alternatively metadata + the crawl data directly in the webhook payload(should be configurable IMHO).

From my understanding:

• current hooks are for dynamically modifying the crawl mid run
• webhooks are for notifying crawl job status

Is something like webhooks planned?

[unclecode]

I got it, you need the crawl4ai server calls the provided webhook url within each of the available hooks, right? Just bear in mind that some of the input parameters wont be available, like playwright Browser, Page or such things, however rest of data you will have access. Does that make sense? @EcksDy

[EcksDy]

@unclecode

Not exactly, in my usecase I've generated a typescript HTTP client and calling the REST API that is available on the docker deployments.

What happens currently:

1. create a crawl job by sending a POST /crawl/job with a CrawlRunConfig, get a task_id
2. poll GET /crawl/job/{task_id} until the job is done and there is data

What I'm looking for from the webhook solution:

1. create a crawl job by sending a POST /crawl/job with a CrawlRunConfig, and additional webhook_url, webhook_data_in_payload options, get a task_id
2. crawl4ai finishes the task, sending a POST request to webhook_url, with the payload: task_id, task_type(llm/md/crawl/etc), status, timestamp of when the task was finished, optionally the crawl data(depending on webhook_data_in_payload)
3. if no data on the payload, handler service will fetch it from GET /crawl/job/{task_id}

This way there's no need for polling, and the handler service can be notified that the crawl job was done and get to work processing the data. Webhooks are usually implemented with an exponential backoff in case there was an error, to try and ensure their arrival.

So no need to send a webhook at every hook, only at the point when the crawl job is done(success/failure).

P.S.
Additionally webhook_url and webhook_data_in_payload can be set on a config level, so all jobs post their finish status(and optionally data) to the specified endpoint.

[unclecode]

@EcksDy

Oh ok, I got it, all the time I was thinking you are talking about the crawler hooks. Ok understand, very useful, we gonna add it soon @ntohidi

[ntohidi]

Hi! We've implemented this feature. Here's the example: https://github.com/unclecode/crawl4ai/blob/main/docs/examples/docker_webhook_example.py