Skip to content

Commit 51a7d94

Browse files
authored
Merge pull request #821 from wanzirong/feature/max-concurrency-param
feat: 添加并发爬虫数量控制参数 --max_concurrency_num
2 parents 94553fd + df39d29 commit 51a7d94

File tree

1 file changed

+9
-0
lines changed

1 file changed

+9
-0
lines changed

cmd_arg/arg.py

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,14 @@ def main(
258258
rich_help_panel="Comment Configuration",
259259
),
260260
] = config.CRAWLER_MAX_COMMENTS_COUNT_SINGLENOTES,
261+
max_concurrency_num: Annotated[
262+
int,
263+
typer.Option(
264+
"--max_concurrency_num",
265+
help="Maximum number of concurrent crawlers",
266+
rich_help_panel="Performance Configuration",
267+
),
268+
] = config.MAX_CONCURRENCY_NUM,
261269
) -> SimpleNamespace:
262270
"""MediaCrawler 命令行入口"""
263271

@@ -283,6 +291,7 @@ def main(
283291
config.SAVE_DATA_OPTION = save_data_option.value
284292
config.COOKIES = cookies
285293
config.CRAWLER_MAX_COMMENTS_COUNT_SINGLENOTES = max_comments_count_singlenotes
294+
config.MAX_CONCURRENCY_NUM = max_concurrency_num
286295

287296
# Set platform-specific ID lists for detail/creator mode
288297
if specified_id_list:

0 commit comments

Comments
 (0)