Skip to content

Commit 19c3af7

Browse files
committed
V4.0.1, update readme
1 parent d28088b commit 19c3af7

File tree

3 files changed

+9
-16
lines changed

3 files changed

+9
-16
lines changed

.gitignore

-7
This file was deleted.

README.md

+9-9
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# PSpider
22

3-
A simple web spider frame written by Python, which needs Python3.6+
3+
A simple web spider frame written by Python, which needs Python3.8+
44

55
### Features of PSpider
66

@@ -18,14 +18,14 @@ A simple web spider frame written by Python, which needs Python3.6+
1818
### Procedure of PSpider
1919

2020
![](procedure.png)
21-
①: Fetchers get url from UrlQueue, and make requests based on this url
22-
②: Put the result(content) of ① to HtmlQueue, and so Parser can get it
23-
③: Parser gets content from HtmlQueue, and parses it to get new urls and item
24-
④: Put the new urls to UrlQueue, and so Fetchers can get it
25-
⑤: Put the item to ItemQueue, and so Saver can get it
26-
⑥: Saver gets item from ItemQueue, and saves it to filesystem or database
27-
⑦: Proxieser gets proxies from web or database, and puts proxies to ProxiesQueue
28-
⑧: Fetcher gets proxies from ProxiesQueue if needed, and makes requests based on this proxies
21+
①: Fetchers get TaskFetch from QueueFetch, and make requests based on this task
22+
②: Put the result(TaskParse) of ① to QueueParse, and so Parser can get task from it
23+
③: Parser gets task from QueueParse, and parses content to get new TaskFetchs and TaskSave
24+
④: Put the new TaskFetchs to QueueFetch, and so Fetchers can get task from it again
25+
⑤: Put the TaskSave to QueueSave, and so Saver can get task from it
26+
⑥: Saver gets TaskSave from QueueSave, and saves items to filesystem or database
27+
⑦: Proxieser gets proxies from web or database, and puts proxies to QueueProxies
28+
⑧: Fetcher gets proxies from QueueProxies if needed, and makes requests based on this proxies
2929

3030
### Tutorials of PSpider
3131

procedure.png

26.1 KB
Loading

0 commit comments

Comments
 (0)