Intelligent Deep Crawl

[Ashoolak]

I'm trying to build a crawler that takes a public websites URL let's say a hospital's researcher directory + a goal for contacts to find (e.g. brain surgeons) as input and outputs the name, title, and email of all the people that match the crawl's goal. This would have to work on any website which makes things extract difficult as I can't just use css selectors (since the structure is unknown.) So to make the cost of the AI extraction not absurd I'm trying implement intelligent crawling. I'm abusing LLM calls to this right now in an effort to replicate how a human assigned such a task on a random website would behave. But was wondering if there's a better way of doing this / if such a thing already exists and I just haven't been able to find it?