Switch to GitHub Proxy

Author: Ra1ny-Yuki

.github/workflows/scheduled_update.yml

@@ -17,7 +17,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: actions/setup-python@v4
         with:
-          python-version: '3.8'
+          python-version: '3.9'
       - uses: actions/cache@v3
         with:
           path: ~/.cache/pip
@@ -50,10 +50,11 @@ jobs:
         run: |
           rm -rf ./meta/.git
           cd ./scripts
+          python main.py remove
           python main.py meta
         env:
           github_api_token: ${{ secrets.TOKEN_FOR_GITHUB_API }}
-          FASTGIT_URL: ${{ vars.FASTGIT_URL }}
+#          FASTGIT_URL: ${{ vars.FASTGIT_URL }}
 
       - name: Deploy meta
         uses: s0/git-publish-subdir-action@develop
@@ -63,7 +64,7 @@ jobs:
           FOLDER: meta
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 #          SSH_PRIVATE_KEY: ${{ secrets.SSH_KEY }}
-#          SQUASH_HISTORY: true
+          SQUASH_HISTORY: true
           SKIP_EMPTY_COMMITS: true
 
       - name: Deploy doc
@@ -74,5 +75,5 @@ jobs:
           FOLDER: catalogue
 #          SSH_PRIVATE_KEY: ${{ secrets.SSH_KEY }}
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-#          SQUASH_HISTORY: true
+          SQUASH_HISTORY: true
           SKIP_EMPTY_COMMITS: true

readme.md

@@ -1,23 +1,20 @@
 **English** | [中文](readme_cn.md)
 
 <h1 align="center"> MCDReforged PluginCatalogue</h1>
-<h3 align="center"><i> (with <a href="https://fastgit.org/">FastGit UK</a>)</i></h3>
+<h3 align="center"><i> (with <a href="https://mirror.ghproxy.com//">GitHub Proxy</a>)</i></h3>
 <p align="center"> 
   A repository for listing <a href="https://github.com/Fallen-Breath/MCDReforged">MCDReforged</a> plugins.
   <br><i>
-  Replaced all the URLs to fastgit in <a href="https://github.com/Lazy-Bing-Server/PluginCatalogue-FG/tree/meta">meta</a> branch to accelerate automatic plugin management
+  Replaced all the download URLs to GitHub in <a href="https://github.com/Lazy-Bing-Server/PluginCatalogue-GP/tree/meta">meta</a> branch to accelerate automatic plugin management
   </i><br>
   <br>
-  <a href="https://github.com/Lazy-Bing-Server/PluginCatalogue-FG/blob/catalogue/readme.md"><strong>Main Plugins Catalogue</strong></a>
+  <a href="https://github.com/Lazy-Bing-Server/PluginCatalogue-GP/blob/catalogue/readme.md"><strong>Main Plugins Catalogue</strong></a>
   ·
   <a href="https://github.com/MCDReforged/PluginCatalogue/blob/legacy/readme.md">Legacy Plugins Catalogue</a> 
   <br>
   <i>Current repository didn't copy the data of original repository and therefore the legacy catalogue link above still directs to the legacy branch of original repository</i>
   <br>
-  <br>
-  <a href="https://hub.fgit.ml/Lazy-Bing-Server/PluginCatalogue-FG">Access this repository with FastGit UK</a>
-  <br>
-  <i>If you are accessing this repo with FastGit, clicking this link will only bring you back here and nothing will happen</i>
+  <i>If you are accessing this repo with GitHub Proxy, clicking this link will only bring you back here and nothing will happen</i>
 </p>
 <br>
 
@@ -32,24 +29,28 @@
 
 If you would like to add your plugin to the plugin repository, you may wish to submit a PR. **Note that these PR should be submitted to [the original repository](https://github.com/MCDReforged/PluginCatalogue).** Check out the [documentation](https://mcdreforged.readthedocs.io/en/latest/plugin_dev/plugin_catalogue.html) for more information
 
-You can learn about the plugin data provided by the plugin repository [here](https://github.com/Lazy-Bing-Server/PluginCatalogue-FG/tree/meta)
+You can learn about the plugin data provided by the plugin repository [here](https://github.com/Lazy-Bing-Server/PluginCatalogue-GP/tree/meta)
 
 ## Plugin Source Configuration
 
 When connection to GitHub is unstable, change the plugin source configuration to this repo for plugin managers to accelerate download. The following are the ways to configure plugin source for plugin managers, if there's any different in configuration to the document of the mentioned plugin manager, please make the description in its own document your primary reference
 
-> ⚠️ <strong>Warning: FastGit service is not absolutely stable. when access to the plugin source configured as following is denied, please restore plugin source config. </strong>
+> ⚠️ <strong>Warning: GitHub Proxy service is not absolutely stable. when access to the plugin source configured as following is denied, please restore plugin source config. </strong>
 
-### [MCDReforged Plugin Manager](https://github.com/Ivan-1F/MCDReforgedPluginManager) *(by [Ivan1F](https://github.com/Ivan-1F))*
+### [MCDReforged](https://mcdreforged.com) official plugin management
 
-Configure the following field in its configuration file `config/mcdreforged_plugin_manager/config.yml` to：
+MCDReforged introduced plugin installation feature in version 2.13
+
+Configure the specified field here in MCDR config file `config.yml` to:
 
 ```commandline
-source: https://hub.fgit.gq/Lazy-Bing-Server/PluginCatalogue-FG/archive/refs/heads/meta.zip
+catalogue_meta_url: https://mirror.ghproxy.com/https://raw.githubusercontent.com/Lazy-Bing-Server/PluginCatalogue-GP/meta/everything.json.xz
 ```
-or
+
+### [MCDReforged Plugin Manager](https://github.com/Ivan-1F/MCDReforgedPluginManager) *(by [Ivan1F](https://github.com/Ivan-1F))*
+
+Configure the following field in its configuration file `config/mcdreforged_plugin_manager/config.yml` to：
 
 ```commandline
-source: https://hub.fgit.ml/Lazy-Bing-Server/PluginCatalogue-FG/archive/refs/heads/meta.zip
+source: https://mirror.ghproxy.com/https://github.com/Lazy-Bing-Server/PluginCatalogue-GP/archive/refs/heads/meta.zip
 ```
-

readme_cn.md

@@ -1,25 +1,20 @@
 [English](readme.md) | **中文**
 
 <h1 align="center">MCDReforged PluginCatalogue</h1>
-<h3 align="center"><i> (使用 <a href="http://fastgit.org/">FastGit UK</a>)</i></h3>
+<h3 align="center"><i> (使用 <a href="https://mirror.ghproxy.com/">GitHub Proxy</a>)</i></h3>
 
 <p align="center">
   一个用于列出 <a href="https://github.com/Fallen-Breath/MCDReforged">MCDReforged</a> 插件的仓库。
   <br>
-  <i><a href="https://github.com/Lazy-Bing-Server/PluginCatalogue-FG/tree/meta">meta</a> 分支中的 URL 已被替换为指向 FastGit 用以加速自动化插件管理</i>
+  <i><a href="https://github.com/Lazy-Bing-Server/PluginCatalogue-GP/tree/meta">meta</a> 分支中的下载 URL 已被替换为指向 GitHub Proxy 用以加速自动化插件管理</i>
   <br>
   <br>
-  <a href="https://github.com/Lazy-Bing-Server/PluginCatalogue-FG/blob/catalogue/readme-zh_cn.md"><strong>主插件目录</strong></a>
+  <a href="https://github.com/Lazy-Bing-Server/PluginCatalogue-GP/blob/catalogue/readme-zh_cn.md"><strong>主插件目录</strong></a>
   ·
   <a href="https://github.com/MCDReforged/PluginCatalogue/blob/legacy/readme_cn.md">旧插件目录</a>
   <br>
   <i>当前仓库未对原插件仓库中的旧插件目录实施镜像，因此上方的旧插件目录链接仍指向原插件仓库的旧插件分支</i>
   <br>
-  <br>
-  <a href="https://hub.fgit.ml/Lazy-Bing-Server/PluginCatalogue-FG">通过 FastGit UK 访问该仓库</a>
-  <br>
-  <i>若当前已经是通过 FastGit 访问，则点击以上链接将回到此处，无事发生</i>
-  <br>
 </p>
 <br>
 
@@ -33,27 +28,30 @@
 
 如果你想添加你的插件到插件仓库中，不妨交个 PR。**请注意这些 PR 应该被提交至[原插件仓库](https://github.com/MCDReforged/PluginCatalogue)**。查看 [文档](https://mcdreforged.readthedocs.io/zh_CN/latest/plugin_dev/plugin_catalogue.html) 以了解更多信息
 
-你可以在 [这里](https://github.com/MCDReforged/PluginCatalogue/tree/meta) 了解到插件仓库提供的插件数据
+你可以在 [这里](https://github.com/Lazy-Bing-Server/PluginCatalogue-GP/tree/meta) 了解到插件仓库提供的插件数据
 
 
 ## 插件源配置
 
 当 GitHub 连接不稳定时，变更插件管理器的插件源配置指向该仓库可加速下载，以下介绍插件管理器中配置插件源的方式，如配置方式与该插件管理器文档存在出入，请首先参考它自己文档的描述
 
-> ⚠️ **注意：FastGit 的服务并非绝对稳定，当无法访问如下配置的插件源时，请还原插件源配置**
+> ⚠️ **注意：GitHub Proxy 的服务并非绝对稳定，当无法访问如下配置的插件源时，请还原插件源配置**
 
-### [MCDReforged Plugin Manager](https://github.com/Ivan-1F/MCDReforgedPluginManager) *(由 [Ivan1F](https://github.com/Ivan-1F) 开发)*
 
+### [MCDReforged](https://mcdreforged.com) 官方插件管理
 
-在其配置文件 `config/mcdreforged_plugin_manager/config.yml` 中调整如下配置: 
+MCDReforged 在 2.13 版本引入了插件安装特性
+
+在其配置文件 `config.yml` 中调整如下配置:
 
 ```commandline
-source: https://hub.fgit.gq/Lazy-Bing-Server/PluginCatalogue-FG/archive/refs/heads/meta.zip
+catalogue_meta_url: https://mirror.ghproxy.com/https://raw.githubusercontent.com/Lazy-Bing-Server/PluginCatalogue-GP/meta/everything.json.xz
 ```
 
-或者是
+### [MCDReforged Plugin Manager](https://github.com/Ivan-1F/MCDReforgedPluginManager) *(由 [Ivan1F](https://github.com/Ivan-1F) 开发)*
+
+在其配置文件 `config/mcdreforged_plugin_manager/config.yml` 中调整如下配置: 
 
 ```commandline
-source: https://hub.fgit.ml/Lazy-Bing-Server/PluginCatalogue-FG/archive/refs/heads/meta.zip
+source: https://mirror.ghproxy.com/https://github.com/Lazy-Bing-Server/PluginCatalogue-GP/archive/refs/heads/meta.zip
 ```
-

scripts/constants.py

@@ -22,6 +22,11 @@
 CATALOGUE_FOLDER = os.path.join(ROOT, 'catalogue')
 META_FOLDER = os.path.join(ROOT, 'meta')
 
+GHPROXY_URL = "https://mirror.ghproxy.com/"
+CREATE_GZ = ["all.json", "authors.json", "everything.json", "everything_slim.json", "plugins.json"]
+CREATE_LZMA = ["everything.json", "everything_slim.json"]
+COMPACT = ["everything.json", "everything_slim.json", "plugins.json"]
+
 PROXIES = None
 if 'http_proxy' in os.environ:
 	http_proxy = os.environ['http_proxy']  # e.g. 127.0.0.1:1082

scripts/main.py

@@ -1,26 +1,71 @@
 import json
 from argparse import ArgumentParser
-from constants import CATALOGUE_FOLDER, META_FOLDER, MAPPING_FILE
+from constants import CATALOGUE_FOLDER, META_FOLDER, GHPROXY_URL, CREATE_LZMA, CREATE_GZ, COMPACT
 from collections import namedtuple
 
 import os
 import typing as ty
 
+import gzip
+import lzma
+
+from contextlib import contextmanager
+
 
 WalkedDir = namedtuple('WalkedDir', 'dirpath dirnames filenames')
 Iterable = ty.Optional[ty.Iterable]
 
 
-def get_fastgit_urls_from_file() -> ty.Dict[str, str]:
-    with open(MAPPING_FILE, 'r', encoding='utf8') as f:
-        return json.load(f)
+@contextmanager
+def open_for_write(file_path: str):
+    """
+    Just like open() in 'w' mode, but create the directory automatically
+    """
+    dir_path = os.path.dirname(file_path)
+    if not os.path.isdir(dir_path):
+        os.makedirs(dir_path)
+    with open(file_path, 'w', encoding='utf8') as file:
+        yield file
+
+
+def save_json(data: dict, file_path: str, *, compact: bool = False, with_gz: bool = False, with_xz: bool = False):
+    if compact:
+        s = json.dumps(data, ensure_ascii=False, separators=(',', ':'))
+    else:
+        s = json.dumps(data, indent=2, ensure_ascii=False)
+
+    with open_for_write(file_path) as f:
+        f.write(s)
+    if with_gz:
+        with gzip.GzipFile(file_path + '.gz', 'wb', mtime=0) as zf:
+            zf.write(s.encode('utf8'))
+    if with_xz:
+        with lzma.open(file_path + '.xz', 'wb', format=lzma.FORMAT_XZ) as xf:
+            xf.write(s.encode('utf8'))
+
+
+def _update(folder, included_suffixes: Iterable = None, excluded_prefixes: Iterable = None):
+    return __walk_n_process(
+        folder, __update,
+        included_suffixes=included_suffixes, excluded_prefixes=excluded_prefixes
+    )
+
+
+def __update(matched: bool, path: str):
+    if not matched:
+        print(f'Excluded {path}')
+        return
+    print(f'Processing {path}...')
 
+    with open(path, 'r', encoding='utf8') as f:
+        data = json.load(f)
+    result_data = recursively_add_ghproxy_prefix(data)
 
-def get_fastgit_urls() -> ty.Dict[str, str]:
-    return json.loads(os.getenv('FASTGIT_URL'))
+    file_name = os.path.basename(path)
+    save_json(result_data, path, compact=file_name in COMPACT, with_gz=file_name in CREATE_GZ, with_xz=file_name in CREATE_LZMA)
 
 
-def __update(folder, included_suffixes: Iterable = None, excluded_prefixes: Iterable = None):
+def __walk_n_process(folder, func: ty.Callable[[bool, str], ty.Any], included_suffixes: Iterable = None, excluded_prefixes: Iterable = None):
     if included_suffixes is None:
         included_suffixes = []
     if excluded_prefixes is None:
@@ -38,37 +83,58 @@ def __update(folder, included_suffixes: Iterable = None, excluded_prefixes: Iter
                     matched = False
 
             path = os.path.join(folder.dirpath, file)
-            if not matched:
-                print(f'Excluded {path}')
-                continue
-
-            print(f'Processing {file}...')
-            to_replace = get_fastgit_urls()
-
-            path = os.path.join(folder.dirpath, file)
-            with open(path, 'r', encoding='utf8') as f:
-                data = f.read()
-
-            for old, new in to_replace.items():
-                data = data.replace(old, new)
-
-            with open(path, 'w', encoding='utf8') as f:
-                f.write(data)
+            func(matched, path)
+
+
+def recursively_add_ghproxy_prefix(data: ty.Union[dict, list, ty.Any]):
+    if isinstance(data, dict):
+        result_dict = {}
+        for k, v in data.items():
+            if k == "browser_download_url" and isinstance(v, str):
+                result_dict[k] = GHPROXY_URL + v
+            else:
+                result_dict[k] = recursively_add_ghproxy_prefix(v)
+        return result_dict
+    elif isinstance(data, list):
+        result_list = []
+        for item in data:
+            result_list.append(recursively_add_ghproxy_prefix(item))
+        return result_list
+    else:
+        return data
 
 
 def update_data():
     print('Processing meta branch')
-    __update(META_FOLDER, included_suffixes=['.json'], excluded_prefixes=['.'])
+    _update(META_FOLDER, included_suffixes=['.json'], excluded_prefixes=['.'])
 
 
 def update_doc():
     print('Processing catalogue branch')
-    __update(CATALOGUE_FOLDER, included_suffixes=['.md'])
+    _update(CATALOGUE_FOLDER, included_suffixes=['.md'])
+
+
+def _remove(folder, included_suffixes: Iterable = None, excluded_prefixes: Iterable = None):
+    __walk_n_process(
+        folder, __remove,
+        included_suffixes=included_suffixes, excluded_prefixes=excluded_prefixes
+    )
+
+
+def __remove(match: bool, path: str):
+    if match:
+        print(f"Removing file {path}")
+        os.remove(path)
+
+
+def remove_meta_archives():
+    print("Remove LZMA and GZ files...")
+    _remove(META_FOLDER, included_suffixes=['.xz', '.gz'])
 
 
 def main():
     parser = ArgumentParser(
-        prog='python manager', description='Plugin Catalogue (FastGit Accelerated) Manager'
+        prog='python manager', description='Plugin Catalogue (GitHub Proxy Accelerated) Manager'
     )
 
     subparsers = parser.add_subparsers(title='Command', help='Available commands', dest='subparser_name')
@@ -78,6 +144,8 @@ def main():
     args = parser.parse_args()
     if args.subparser_name == 'meta':
         update_data()
+    elif args.subparser_name == 'remove':
+        remove_meta_archives()
     elif args.subparser_name == 'doc':
         update_doc()