Question about Multi-threaded Link Checker task #171
Description
Hi. I have a difficulty understanding the following subtask:
Use threads to check the links in parallel: send the URLs to be checked to a channel and let a few threads check the URLs in parallel.
Is the suggestion here to create a channel on the main thread, which will spawn a new thread whenever it receives a URL and then run a recursive link checker for this URL on the spawned thread? So for instance, a user could simultaneously run two parallel link checkers for site A and site B.
Or is it more about parallelizing the recursive checks of the same site? So for instance, you start with the start page of site A, extract a list of URLs from that page and send them to some queue from which several other threads can pick up URLs, process them and add new extracted URLs to the queue, and so on.