Merge pull request #58 from CESNET/develop

Merge develop 1.1.1 into Main

Author: jirivrany

.github/workflows/python-app.yml

@@ -1,28 +1,25 @@
-# This workflow will install Python dependencies, run tests and lint with a single version of Python
+# This workflow will install Python dependencies, run tests and lint with multiple versions of Python
 # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
-
 name: Python application
-
 on:
   push:
     branches: [ "master", "develop" ]
   pull_request:
     branches: [ "master", "develop" ]
-
 permissions:
   contents: read
-
 jobs:
   build:
-
     runs-on: ubuntu-latest
-
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.10", "3.11", "3.12"]
     steps:
     - uses: actions/checkout@v3
-    - name: Set up Python 3.9
+    - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v3
       with:
-        python-version: "3.9"
+        python-version: ${{ matrix.python-version }}
     - name: Setup timezone
       uses: zcong1993/setup-timezone@master
       with:

README.md

@@ -35,26 +35,39 @@ See how is ExaFS integrated into the network in the picture below.
 ## System overview
 
 ![ExaFS schema](./docs/app_schema_en.png)
+The core component of ExaFS is a web application written in Python using the Flask framework. It provides a user interface for managing ExaBGP rules (CRUD operations) and also exposes a REST API with similar functionality. The web application uses Shibboleth for authentication, while the REST API relies on token-based authentication.
 
-The central part of the ExaFS is a web application, written in Python3.6 with Flask framework. It provides a user interface for ExaBGP rule CRUD operations. The application also provides the REST API with CRUD operations for the configuration rules. The web app uses Shibboleth authorization; the REST API is using token-based authorization. 
+The application generates ExaBGP commands and forwards them to the ExaBGP process. All rules are thoroughly validated—only valid rules are stored in the database and sent to the ExaBGP connector.
 
-The app creates the ExaBGP commands and forwards them to ExaBGP process. All rules are carefully validated, and only valid rules are stored in the database and sent to the ExaBGP connector.
- 
-This second part of the system is another application that replicates the received command to the stdout. The connection between ExaBGP daemon and stdout of ExaAPI (ExaBGP process) is specified in the ExaBGP config. 
+The second component of the system is a separate application that replicates received commands to `stdout`. The connection between the ExaBGP daemon and the `stdout` of the ExaAPI (ExaBGP process) is defined in the ExaBGP configuration.
 
-This API was a part of the project, but now has been moved to own repository. You can use [pip package exabgp-process](https://pypi.org/project/exabgp-process/) or clone the git repo. Or you can create your own version.
- 
-Every time this process gets a command from ExaFS, it replicates this command to the ExaBGP service through the stdout. The registered service then updates the ExaBGP table – create, modify or remove the rule from command.
+This API was originally part of the same project but has since been moved to its own repository. You can use the [exabgp-process pip package](https://pypi.org/project/exabgp-process/), clone the Git repository, or develop your own implementation.
 
-You may also need to monitor the ExaBGP and renew the commands after restart / shutdown. In docs you can find and example of system service named Guarda. This systemctl service is running in the host system and gets a notification on each restart of ExaBGP service via systemctl WantedBy config option. For every restart of ExaBGP the Guarda service will put all the valid and active rules to the ExaBGP rules table again.
+Each time this process receives a command from ExaFS, it outputs it to `stdout`, allowing the ExaBGP service to process the command and update its routing table—creating, modifying, or removing rules accordingly.
+
+It may also be necessary to monitor ExaBGP and re-announce rules after a restart or shutdown. This can be handled via the ExaBGP service configuration, or by using an example system service called **Guarda**, described in the documentation. In either case, the key mechanism is calling the application endpoint `/rules/announce_all`. This endpoint is only accessible from `localhost`; a local IP address must be configured in the application settings.
 
 ## DOCS
+### Instalation related
+* [ExaFS Ansible deploy](https://github.com/CESNET/ExaFS-deploy) - repository with Ansbile playbook for deploying ExaFS with Docker Compose. 
 * [Install notes](./docs/INSTALL.md)
-* [API documentation ](https://exafs.docs.apiary.io/#)
 * [Database backup configuration](./docs/DB_BACKUP.md)
 * [Local database instalation notes](./docs/DB_LOCAL.md)
+### API
+The REST API is documented using Swagger (OpenAPI). After installing and running the application, the API documentation is available locally at the /apidocs/ endpoint. This interactive documentation provides details about all available endpoints, request and response formats, and supported operations, making it easier to integrate and test the API.
+
+
 
 ## Change Log
+- 1.1.1 - Machine API Key rewrited. 
+    - API keys for machines are now tied to one of the existing users. If there is a need to have API access for machine, first create service user, and set the access rights. Then create machine key as Admin and assign it to this user. 
+- 1.1.0 - Major Architecture Refactoring and Whitelist Integration
+    - Code Organization and Architecture Improvements. Significant architectural refactoring focused on better separation of concerns and improved maintainability. The most notable change is the introduction of a dedicated **services layer** that extracts business logic from view controllers. Key service modules include `rule_service.py` for rule management operations, `whitelist_service.py` for whitelist functionality, and `whitelist_common.py` for shared whitelist utilities. 
+    - The **models structure** has been reorganized with better separation into logical modules. Rule models are now organized under `flowapp/models/rules/` with separate files for different rule types (`flowspec.py`, `rtbh.py`, `whitelist.py`), while maintaining backward compatibility through the main models `__init__.py`. Form handling has also been improved with better organization under `flowapp/forms/` and enhanced validation logic.
+    - **RTBH Whitelist Integration** This system automatically evaluates new RTBH rules against existing whitelists and can automatically modify or block rules that conflict with whitelisted networks. When an RTBH rule is created that intersects with a whitelist entry, the system can:
+        - **Automatically whitelist** rules that exactly match or are contained within whitelisted networks
+        - **Create subnet rules** when RTBH rules are supersets of whitelisted networks, automatically generating the non-whitelisted portions
+        - **Maintain rule cache** that tracks relationships between rules and whitelists for proper cleanup
 - 1.0.2 - fixed bug in IPv6 Flowspec messages
 - 1.0.1 . minor bug fixes
 - 1.0.0 . Major changes

docs/guarda-service/README.md

@@ -1,5 +1,4 @@
 # Guarda Service for ExaBGP
-
 This is a systemd service designed to monitor ExaBGP and reapply commands after a restart or shutdown. The guarda.service runs on the host system and is triggered whenever the ExaBGP service restarts, thanks to the WantedBy configuration in systemd. After each restart, the Guarda service will reapply all valid and active rules to the ExaBGP rules table.
 
 ## Usage (as root)

flowapp/__about__.py

@@ -1 +1 @@
-__version__ = "1.0.2"
+__version__ = "1.1.1"

flowapp/__init__.py

@@ -1,10 +1,6 @@
 # -*- coding: utf-8 -*-
-import babel
-import logging
-from loguru import logger
+from flask import Flask, redirect, render_template, session, url_for
 
-from flask import Flask, redirect, render_template, session, url_for, request
-from flask.logging import default_handler
 from flask_sso import SSO
 from flask_sqlalchemy import SQLAlchemy
 from flask_wtf.csrf import CSRFProtect
@@ -26,16 +22,16 @@
 swagger = Swagger(template_file="static/swagger.yml")
 
 
-class InterceptHandler(logging.Handler):
-
-    def emit(self, record):
-        logger_opt = logger.opt(depth=6, exception=record.exc_info, colors=True)
-        logger_opt.log(record.levelname, record.getMessage())
-
-
 def create_app(config_object=None):
     app = Flask(__name__)
 
+    # Load the default configuration for dashboard and main menu
+    app.config.from_object(InstanceConfig)
+    if config_object:
+        app.config.from_object(config_object)
+
+    app.config.setdefault("VERSION", __version__)
+
     # SSO configuration
     SSO_ATTRIBUTE_MAP = {
         "eppn": (True, "eppn"),
@@ -47,13 +43,6 @@ def create_app(config_object=None):
     migrate.init_app(app, db)
     csrf.init_app(app)
 
-    # Load the default configuration for dashboard and main menu
-    app.config.from_object(InstanceConfig)
-    if config_object:
-        app.config.from_object(config_object)
-
-    app.config.setdefault("VERSION", __version__)
-
     # Init SSO
     ext.init_app(app)
 
@@ -64,76 +53,28 @@ def create_app(config_object=None):
     if app.config.get("BEHIND_PROXY", False):
         app.wsgi_app = ProxyFix(app.wsgi_app, x_for=1, x_proto=1, x_host=1, x_prefix=1)
 
-    from flowapp import models, constants, validators
-    from .views.admin import admin
-    from .views.rules import rules
-    from .views.api_v1 import api as api_v1
-    from .views.api_v2 import api as api_v2
-    from .views.api_v3 import api as api_v3
-    from .views.api_keys import api_keys
+    from flowapp import models, constants
     from .auth import auth_required
-    from .views.dashboard import dashboard
-
-    # no need for csrf on api because we use JWT
-    csrf.exempt(api_v1)
-    csrf.exempt(api_v2)
-    csrf.exempt(api_v3)
-
-    app.register_blueprint(admin, url_prefix="/admin")
-    app.register_blueprint(rules, url_prefix="/rules")
-    app.register_blueprint(api_keys, url_prefix="/api_keys")
-    app.register_blueprint(api_v1, url_prefix="/api/v1")
-    app.register_blueprint(api_v2, url_prefix="/api/v2")
-    app.register_blueprint(api_v3, url_prefix="/api/v3")
-    app.register_blueprint(dashboard, url_prefix="/dashboard")
-
-    # register loguru as handler
-    app.logger.removeHandler(default_handler)
-    app.logger.addHandler(InterceptHandler())
-
-    @ext.login_handler
-    def login(user_info):
-        try:
-            uuid = user_info.get("eppn")
-        except KeyError:
-            uuid = False
-            return render_template("errors/401.html")
 
-        return _handle_login(uuid)
+    # Register blueprints
+    from .utils import register_blueprints
 
-    @app.route("/logout")
-    def logout():
-        session["user_uuid"] = False
-        session["user_id"] = False
-        session.clear()
-        return redirect(app.config.get("LOGOUT_URL"))
+    register_blueprints(app, csrf)
 
-    @app.route("/ext-login")
-    def ext_login():
-        header_name = app.config.get("AUTH_HEADER_NAME", "X-Authenticated-User")
-        if header_name not in request.headers:
-            return render_template("errors/401.html")
+    # configure logging
+    from .utils import configure_logging
 
-        uuid = request.headers.get(header_name)
-        if not uuid:
-            return render_template("errors/401.html")
+    configure_logging(app)
 
-        return _handle_login(uuid)
+    # register error handlers
+    from .utils import register_error_handlers
 
-    @app.route("/local-login")
-    def local_login():
-        print("Local login started")
-        if not app.config.get("LOCAL_AUTH", False):
-            print("Local auth not enabled")
-            return render_template("errors/401.html")
+    register_error_handlers(app)
 
-        uuid = app.config.get("LOCAL_USER_UUID", False)
-        if not uuid:
-            print("Local user not set")
-            return render_template("errors/401.html")
+    # register auth handlers
+    from .utils import register_auth_handlers
 
-        print(f"Local login with {uuid}")
-        return _handle_login(uuid)
+    register_auth_handlers(app, ext)
 
     @app.route("/")
     @auth_required
@@ -192,89 +133,10 @@ def select_org(org_id=None):
     def shutdown_session(exception=None):
         db.session.remove()
 
-    # HTTP error handling
-    @app.errorhandler(404)
-    def not_found(error):
-        return render_template("errors/404.html"), 404
-
-    @app.errorhandler(500)
-    def internal_error(exception):
-        app.logger.exception(exception)
-        return render_template("errors/500.html"), 500
-
-    @app.context_processor
-    def utility_processor():
-        def editable_rule(rule):
-            if rule:
-                validators.editable_range(rule, models.get_user_nets(session["user_id"]))
-                return True
-            return False
-
-        return dict(editable_rule=editable_rule)
-
-    @app.context_processor
-    def inject_main_menu():
-        """
-        inject main menu config to templates
-        used in default template to create main menu
-        """
-        return {"main_menu": app.config.get("MAIN_MENU")}
-
-    @app.context_processor
-    def inject_dashboard():
-        """
-        inject dashboard config to templates
-        used in submenu dashboard to create dashboard tables
-        """
-        return {"dashboard": app.config.get("DASHBOARD")}
-
-    @app.template_filter("strftime")
-    def format_datetime(value):
-        if value is None:
-            return app.config.get("MISSING_DATETIME_MESSAGE", "Never")
-
-        format = "y/MM/dd HH:mm"
-        return babel.dates.format_datetime(value, format)
-
-    @app.template_filter("unlimited")
-    def unlimited_filter(value):
-        return "unlimited" if value == 0 else value
-
-    def _handle_login(uuid: str):
-        """
-        handles rest of login process
-        """
-        multiple_orgs = False
-        try:
-            user, multiple_orgs = _register_user_to_session(uuid)
-        except AttributeError as e:
-            app.logger.exception(e)
-            return render_template("errors/401.html")
-
-        if multiple_orgs:
-            return redirect(url_for("select_org", org_id=None))
+    # register context processors and template filters
+    from .utils import register_context_processors, register_template_filters
 
-        # set user org to session
-        user_org = user.organization.first()
-        session["user_org"] = user_org.name
-        session["user_org_id"] = user_org.id
-
-        return redirect("/")
-
-    def _register_user_to_session(uuid: str):
-        print(f"Registering user {uuid} to session")
-        user = db.session.query(models.User).filter_by(uuid=uuid).first()
-        print(f"Got user {user} from DB")
-        session["user_uuid"] = user.uuid
-        session["user_email"] = user.uuid
-        session["user_name"] = user.name
-        session["user_id"] = user.id
-        session["user_roles"] = [role.name for role in user.role.all()]
-        session["user_role_ids"] = [role.id for role in user.role.all()]
-        roles = [i > 1 for i in session["user_role_ids"]]
-        session["can_edit"] = True if all(roles) and roles else []
-        # check if user has multiple organizations and return True if so
-        print(f"DEBUG SESSION {session}")
-        return user, len(user.organization.all()) > 1
+    register_context_processors(app)
+    register_template_filters(app)
 
     return app

flowapp/constants.py

@@ -2,6 +2,7 @@
 This module contains constant values used in application
 """
 
+from enum import Enum
 from operator import ge, lt
 
 DEFAULT_SORT = "expires"
@@ -59,7 +60,12 @@
 FORM_TIME_PATTERN = "%Y-%m-%dT%H:%M"
 
 
-class RuleTypes:
+class RuleTypes(Enum):
     RTBH = 1
     IPv4 = 4
     IPv6 = 6
+
+
+class RuleOrigin(Enum):
+    USER = 1
+    WHITELIST = 2