Introduction of PyFunceble v3.3.0 (Teal Blauwbok: Centipede)

Fixed:

- Typos.
- Badges URL.
- Issues under macOS while using the `--multiprocess` argument.

Improved:

- Some sections of the documentation.
- The standard sorting of file content.
- The hierarchical sorting of file content.
- The cleanup logic behind the `--clean-all` argument.
- The way we fix (`/` always at the end of directories)
  the internal paths.
- The timeout of the DNS resolver set to 2 seconds (explicitly)
  to avoid an overwriting from the upstream.
- The mining methodology.
- Database (MySQL/MariaDB) connection methodology.
- Database (MySQL/MariaDB) connection closing methodology.

New:

- SQLAlchemy instead of RAW SQL.
- Automatic migration routine from old (RAW SQL) to new (SQLAlchemy)
  table layout.
- `*.wix.com` special rule.
- `*.hpg.com.br` special rule.
- `*.000webhostapp.com` special rule.
- `*.angelfire.com` special rule.
- Position tracker to jump as quickly as possible
  while resuming from a previous instance.
- Mining (`--mining`) capabilities from a normal/simple test.
- CI tests are running under Python 3.8.5, 3.7.8 and 3.6.11.
- Multiprocessing will be deactivated if less than 2 processes are
  allocated/allowed.
- `--all` argument prints (really) everything.
- Introduction of the new `--store-whois` argument and logic.
  complete WHOIS records are not stored anymore.
- Ensures that we retry the download of the files 3 times before
  stopping completely.
- The `PYFUNCEBLE_DEBUG` and `PYFUNCEBLE_DEBUG_ON_SCREEN` environment variables.

Removed:

- The `db_types` directory.

Contributors:

- @spirillen

.PyFunceble_production.yaml

@@ -282,11 +282,11 @@ http_codes:
 links:
   api_date_format: "https://pyfunceble.funilrys.com/api/date-format"
   api_no_referer: "https://pyfunceble.funilrys.com/api/no-referer"
-  config: "https://raw.githubusercontent.com/funilrys/PyFunceble/dev/.PyFunceble_production.yaml"
-  dir_structure: "https://raw.githubusercontent.com/funilrys/PyFunceble/dev/dir_structure_production.json"
+  config: "https://raw.githubusercontent.com/funilrys/PyFunceble/master/.PyFunceble_production.yaml"
+  dir_structure: "https://raw.githubusercontent.com/funilrys/PyFunceble/master/dir_structure_production.json"
   iana: "https://raw.githubusercontent.com/PyFunceble/iana/master/iana-domains-db.json"
   ipv4_reputation: "https://reputation.alienvault.com/reputation.data"
   psl: "https://raw.githubusercontent.com/PyFunceble/public-suffix/master/public-suffix.json"
   repo: "https://github.com/funilrys/PyFunceble"
-  requirements: "https://raw.githubusercontent.com/funilrys/PyFunceble/dev/requirements.txt"
+  requirements: "https://raw.githubusercontent.com/funilrys/PyFunceble/master/requirements.txt"
   user_agents: "https://raw.githubusercontent.com/PyFunceble/user_agents/master/user_agents.json"

PyFunceble/__init__.py

@@ -25,7 +25,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/abstracts/__init__.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/abstracts/infrastructure.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/
@@ -103,7 +103,7 @@ class Infrastructure:
     :type: str
     """
 
-    PROD_CONFIG_LINK = "https://raw.githubusercontent.com/funilrys/PyFunceble/dev/.PyFunceble_production.yaml"  # pylint: disable=line-too-long
+    PROD_CONFIG_LINK = "https://raw.githubusercontent.com/funilrys/PyFunceble/master/.PyFunceble_production.yaml"  # pylint: disable=line-too-long
     """
     Sets the link to the production configuration file.
 

PyFunceble/abstracts/package.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/
@@ -64,7 +64,7 @@ class Package:
     :type: str
     """
 
-    VERSION = "3.2.14.dev (Teal Blauwbok: Bombardier)"
+    VERSION = "3.3.0. (Teal Blauwbok: Centipede)"
     """
     Sets the package version.
 

PyFunceble/abstracts/platform.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/check.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/cli/__init__.py

@@ -27,7 +27,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/
@@ -97,7 +97,7 @@ def tool():  # pragma: no cover pylint: disable=too-many-branches,too-many-state
 
                 epilog = (
                     f"{Style.BRIGHT}{Fore.YELLOW}For an in-depth usage, explanation and examples of the arguments, "
-                    f"you should read the documentation at{Fore.GREEN} https://pyfunceble.readthedocs.io/en/dev/"
+                    f"you should read the documentation at{Fore.GREEN} https://pyfunceble.readthedocs.io/en/master/"
                     f"{Style.RESET_ALL}\n\n"
                     f"Crafted with {Fore.RED}♥{Fore.RESET} by "
                     f"{Style.BRIGHT}{Fore.CYAN}Nissar Chababy (@funilrys){Style.RESET_ALL} "

PyFunceble/cli/dispatcher.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/cli/execution_time.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/cli/iana.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/cli/production.py

@@ -27,7 +27,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/
@@ -445,7 +445,11 @@ def _update_docs(self, file_to_update):
                 "/en/%s" % "dev": r"en\/%s" % "master",
                 "/pyfunceble-%s.png" % "dev": r"\/pyfunceble-dev.png",
                 "/project/pyfunceble-%s" % "dev": r"\/project\/pyfunceble$",
-                "/badge/pyfunceble-%s" % "dev": r"\/badge\/pyfunceble$",
+                "/badge/pyfunceble-%s\\1"
+                % "dev": r"\/badge\/pyfunceble(/month|/week|)$",
+                "/blob/%s/" % "dev": r"\/blob\/%s\/" % "master",
+                "/pypi/v/pyfunceble-%s.png" % "dev": r"\/pypi\/v\/pyfunceble\.png$",
+                "/\\1/%s/" % "dev": r"\/(logo|graphmls|gifs\/raw)\/%s\/" % "master",
             }
         elif self.is_master_version():
             # The current version is the master version.
@@ -457,7 +461,11 @@ def _update_docs(self, file_to_update):
                 "/en/%s" % "master": r"en\/%s" % "dev",
                 "/pyfunceble-dev.png": r"\/pyfunceble-%s.png" % "dev",
                 "/project/pyfunceble": r"/project\/pyfunceble-%s$" % "dev",
-                "/badge/pyfunceble": r"/badge\/pyfunceble-%s$" % "dev",
+                "/badge/pyfunceble\\1": r"/badge\/pyfunceble-%s(/month|/week|)$"
+                % "dev",
+                "/blob/%s/" % "master": r"\/blob\/%s\/" % "dev",
+                "/pypi/v/pyfunceble.png": r"\/pypi\/v\/pyfunceble-%s\.png$" % "dev",
+                "/\\1/%s/" % "master": r"\/(logo|graphmls|gifs\/raw)\/%s\/" % "dev",
             }
         else:
             # The current version is not the master nor the dev version.

PyFunceble/cli/publicsuffix.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/config/__init__.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/config/loader.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/config/preset.py

@@ -27,7 +27,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/converter/__init__.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/converter/adblock.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/converter/base.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/converter/digit2digits.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/converter/file.py

@@ -27,7 +27,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/converter/internal_url.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/converter/month.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/core/__init__.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/core/api.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/core/cli.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/
@@ -409,7 +409,7 @@ def logs_sharing(cls):
             print(Fore.GREEN + Style.BRIGHT + "You are sharing your logs!")
             print(
                 Fore.MAGENTA + Style.BRIGHT + "Please find more about it at "
-                "https://pyfunceble.readthedocs.io/en/dev/logs-sharing.html !"
+                "https://pyfunceble.readthedocs.io/en/master/logs-sharing.html !"
             )
 
     @classmethod
@@ -421,7 +421,7 @@ def get_upstream_version_file(cls):  # pragma: no cover
         # We initiate the link to the upstream version file.
         # It is hard coded because we may not have the chance to have the
         # configuration file everytime we need it.
-        upstream_link = "https://raw.githubusercontent.com/funilrys/PyFunceble/dev/version.yaml"  # pylint: disable=line-too-long
+        upstream_link = "https://raw.githubusercontent.com/funilrys/PyFunceble/master/version.yaml"  # pylint: disable=line-too-long
 
         upstream_link = PyFunceble.converter.InternalUrl(upstream_link).get_converted()
 

PyFunceble/core/file.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/core/multiprocess.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/core/simple.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/data/__init__.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/data/alembic/__init__.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/data/alembic/env.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/data/alembic/versions/__init__.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/database/__init__.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/

PyFunceble/database/inactive.py

@@ -26,7 +26,7 @@
     https://github.com/funilrys/PyFunceble
 
 Project documentation:
-    https://pyfunceble.readthedocs.io/en/dev/
+    https://pyfunceble.readthedocs.io/en/master/
 
 Project homepage:
     https://pyfunceble.github.io/