Develop the new client

[vikman90]

Parent issue:

• Agent/server communication protocol wazuh#22677

Description

Following the completion of the spike on agent-manager communication, the next step is to implement a functional client for this new protocol. This client will establish a fully functional communication and handshake with the server, transmit queue information, and leave the interface open for the addition of other modules.

Functional requirements

1. Connect to the server.
2. Register and obtain a UUID.
3. Authenticate and acquire a token.
4. Send data using that token. Let such data persist in the agent until it receives a 200 code from the server.
   a. Stateful.
   b. Stateless.
5. Query commands that the server needs to send to the agent.
6. Resilience: reconnect to the server if the connection is lost.
7. Multitasking: support multiple concurrent connections.
8. Read options from a configuration file.
9. Work alongside the queue system.

Implementation restrictions

1. We will adhere to the restrictions declared in issue New Agent comms API endpoint client #1.
2. The configuration will be in TOML format.
3. The configuration will be parsed in a separate component, such as ConfigParser.
4. Support for multiple communications will preferably be implemented without multithreading, using coroutines or similar.
5. If possible, use C++20.

Plan

1. Start from the PoC developed in issue New Agent comms API endpoint client #1.
2. Agree on the queue interface: Agent stateful modules redesign #2.
3. Agree on the command manager interface: Agent command manager #4.
4. Investigate if all target platforms support C++20 with GCC and Clang.
5. Design and implement a task manager Design and development of the concurrency control module for the new agent #24.
6. Design and implement a communicator module Design and development of the communicator component for the new agent #25.
7. Design and implement a configuration parser Design and development of the configuration parser component for the new agent #26.
8. Integrate with the Queue component Integrate the new client with the Queue component #57.
9. Address technical debt Address technical debt for the new agent MVP #64.
10. If necessary, implement a "Dummy" module for testing purposes.

[aritosteles]

Compatibility with gcc 10

The following OS have been tested and they either have a package for gcc10 or gcc10 has been built successfully.

• Debian 10
• Ubuntu 16.04
• Amazon Linux 1
• openSUSE 15
• SUSE 15 - there are packages for gcc12 and gcc13
• MacOS 14
• AlmaLinux 8
• Rocky Linux 8
• RHEL 6
• CentOS 6

[vikman90]

C++20 concepts vs inheritance

I've conducted a trial to model a module pool in C++, aiming to create a class capable of managing references to all modules, starting them, and facilitating communication.

To achieve this, I explored two approaches:

• Inheritance: Introducing an IRunnable interface where modules inherit and implement it.
• Concepts: Utilizing a Runnable concept, with the pool restricting parameters to this concept.

Initially, conceptual constraints seemed preferable over inheritance to ensure each module remains independent of the "Module" definition. However, I noted that using a std::vector<std::any> container — akin to the classic void*[] in C — introduces excessive generality. This approach necessitates runtime type inference for each module access operation, thereby:

• Missing compile-time error detection.
• Incurring slight runtime overhead for type inference, contrary to our aim of avoiding vtables from inheritance.

Moreover, relying on concepts does not entirely decouple modules from the Agent component, as they still need to interact via the Pool if inter-module communication is desired.

Code

inheritance.cpp

#include <iostream>
#include <vector>
#include <memory>

class IRunnable {
public:
    virtual void run() = 0;
    virtual ~IRunnable() = default;
};

class LogCollector : public IRunnable {
public:
    void run() override {
        std::cout << "LogCollector is running" << std::endl;
    }
};

class FIM : public IRunnable {
public:
    void run() override {
        std::cout << "FIM is running" << std::endl;
    }
};

class Pool {
public:
    void addRunnable(std::shared_ptr<IRunnable> runnable) {
        runnables.push_back(runnable);
    }

    void executeAll() {
        for (auto& runnable : runnables) {
            runnable->run();
        }
    }

private:
    std::vector<std::shared_ptr<IRunnable>> runnables;
};

int main() {
    auto logcollector = std::make_shared<LogCollector>();
    auto fim = std::make_shared<FIM>();

    Pool pool;
    pool.addRunnable(logcollector);
    pool.addRunnable(fim);

    pool.executeAll();

    return 0;
}

concept.cpp

#include <iostream>
#include <vector>
#include <any>
#include <type_traits>
#include <concepts>

template<typename T>
concept Runnable = requires(T t) {
    { t.run() } -> std::same_as<void>;
};

class LogCollector {
public:
    void run() {
        std::cout << "LogCollector is running" << std::endl;
    }
};

class FIM {
public:
    void run() {
        std::cout << "FIM is running" << std::endl;
    }
};

class Pool {
public:
    template<Runnable T>
    void addRunnable(T obj) {
        runnables.push_back(std::make_any<T>(std::move(obj)));
    }

    void executeAll() {
        for (auto& a : runnables) {
            try {
                if (a.type() == typeid(LogCollector)) {
                    std::any_cast<LogCollector&>(a).run();
                } else if (a.type() == typeid(FIM)) {
                    std::any_cast<FIM&>(a).run();
                } else {
                    std::cerr << "ERROR: Incompatible element." << std::endl;
                }
            } catch (const std::bad_any_cast&) {
                std::cerr << "ERROR: Incompatible element." << std::endl;
            }
        }
    }

private:
    std::vector<std::any> runnables;
};

int main() {
    Pool pool;

    LogCollector logcollector;
    FIM fim;

    pool.addRunnable(logcollector);
    pool.addRunnable(fim);

    pool.executeAll();

    return 0;
}

Conclusion

	Inheritance	Concepts
Pros	Simpler container setup. Agent independence from individual module definitions.	Module definition detached from Agent during instantiation.
Cons	Each module inherits Module, causing vtable overhead. Dependency of Agent on each module.	Agent dependency on each module persists.

If my analysis holds true, IMHO, I'm inclined towards using inheritance.

[jr0me]

Update: Compatibility with gcc 10

• Fedora 40 🟢

The process to compile gcc10 on Fedora 40 was not straight forward, gcc 10.5 worked but not previous versions. It was necessary to compile GCC with options --enable-version-specific-runtime-libs. Then the example code with coroutines also needed the additional option -static-libgcc as the linker stage failed to find libgcc.

GCC14 is available out of the box nonetheless.

[vikman90]

C++20 concepts vs inheritance (update)

I have further developed a new proposal for the approach using concepts: I introduced wrappers for each module. The addModule() function, which creates the wrappers, is a template function that uses concepts. This way, we achieve decoupling each module class from the module definition.

However, in this proposal, each module must receive a Configuration object to establish its configuration. This object must be created by some configuration parser.

Code

Below, I present the two updated approaches with exactly the same behavior:

inheritance.cpp using an abstract base class

#include <iostream>
#include <map>
#include <memory>
#include <string>

using namespace std;

class Configuration { };

struct IModule {
public:
    virtual ~IModule() = default;
    virtual void run() = 0;
    virtual void stop() = 0;
    virtual int setup(const Configuration& config) = 0;
    virtual string command(const string & query) = 0;
    virtual string name() const = 0;
};

/******************************************************************************/

struct Logcollector : public IModule {
    void run() {
        cout << "+ [Logcollector] is running" << endl;
    }

    int setup(const Configuration & config) {
        return 0;
    }

    void stop() {
        cout << "- [Logcollector] stopped" << endl;
    }

    string command(const string & query) {
        cout << "  [Logcollector] query: " << query << endl;
        return "OK";
    }

    string name() const {
        return "logcollector";
    }
};

struct FIM : public IModule {
    void run() {
        cout << "+ [FIM] is running" << endl;
    }

    int setup(const Configuration & config) {
        return 0;
    }

    void stop() {
        cout << "- [FIM] stopped" << endl;
    }

    string command(const string & query) {
        cout << "  [FIM] query: " << query << endl;
        return "OK";
    }

    string name() const {
        return "fim";
    }
};

struct Inventory : public IModule {
    void run() {
        cout << "+ [Inventory] is running" << endl;
    }

    int setup(const Configuration & config) {
        return 0;
    }

    void stop() {
        cout << "- [Inventory] stopped" << endl;
    }

    string command(const string & query) {
        cout << "  [Inventory] query: " << query << endl;
        return "OK";
    }

    string name() const {
        return "inventory";
    }
};

struct SCA : public IModule {
    void run() {
        cout << "+ [SCA] is running" << endl;
    }

    int setup(const Configuration & config) {
        return 0;
    }

    void stop() {
        cout << "- [SCA] stopped" << endl;
    }

    string command(const string & query) {
        cout << "  [SCA] query: " << query << endl;
        return "OK";
    }

    string name() const {
        return "sca";
    }
};

/******************************************************************************/

class Pool {
public:
    Pool() {
        addModule(make_shared<Logcollector>());
        addModule(make_shared<FIM>());
        addModule(make_shared<Inventory>());
        addModule(make_shared<SCA>());
    }

    void addModule(shared_ptr<IModule> module) {
        modules[module->name()] = module;
    }

    shared_ptr<IModule> getModule(const string & name) {
        return modules.at(name);
    }

    void start() {
        for (const auto &[_, module] : modules) {
            module->run();
        }
    }

    void setup(const Configuration & config) {
        for (const auto &[_, module] : modules) {
            module->setup(config);
        }
    }

    void stop() {
        for (const auto &[_, module] : modules) {
            module->stop();
        }
    }

private:
    map<string, shared_ptr<IModule>> modules;
};

int main() {
    Pool pool;
    Configuration config;

    pool.start();
    pool.setup(config);
    cout << endl;

    try {
        auto logcollector = pool.getModule("logcollector");
        logcollector->command("Hello World!");
    } catch (const out_of_range & e) {
        cerr << "!  OOPS: Module not found." << endl;
    }

    cout << endl;
    pool.stop();

    return 0;
}

wrapper.cpp using a wrapper and concepts

#include <iostream>
#include <functional>
#include <map>
#include <memory>
#include <string>

using namespace std;

class Configuration { };

/******************************************************************************/

struct Logcollector {
    void run() {
        cout << "+ [Logcollector] is running" << endl;
    }

    int setup(const Configuration & config) {
        return 0;
    }

    void stop() {
        cout << "- [Logcollector] stopped" << endl;
    }

    string command(const string & query) {
        cout << "  [Logcollector] query: " << query << endl;
        return "OK";
    }

    string name() const {
        return "logcollector";
    }
};

struct FIM {
    void run() {
        cout << "+ [FIM] is running" << endl;
    }

    int setup(const Configuration & config) {
        return 0;
    }

    void stop() {
        cout << "- [FIM] stopped" << endl;
    }

    string command(const string & query) {
        cout << "  [FIM] query: " << query << endl;
        return "OK";
    }

    string name() const {
        return "fim";
    }
};

struct Inventory {
    void run() {
        cout << "+ [Inventory] is running" << endl;
    }

    int setup(const Configuration & config) {
        return 0;
    }

    void stop() {
        cout << "- [Inventory] stopped" << endl;
    }

    string command(const string & query) {
        cout << "  [Inventory] query: " << query << endl;
        return "OK";
    }

    string name() const {
        return "inventory";
    }
};

struct SCA {
    void run() {
        cout << "+ [SCA] is running" << endl;
    }

    int setup(const Configuration & config) {
        return 0;
    }

    void stop() {
        cout << "- [SCA] stopped" << endl;
    }

    string command(const string & query) {
        cout << "  [SCA] query: " << query << endl;
        return "OK";
    }

    string name() const {
        return "sca";
    }
};

/******************************************************************************/

template<typename T>
concept Module = requires(T t, const Configuration & config, const string & query) {
    { t.run() } -> same_as<void>;
    { t.setup(config) } -> same_as<int>;
    { t.stop() } -> same_as<void>;
    { t.command(query) } -> same_as<string>;
    { t.name() } -> same_as<string>;
};

struct ModuleWrapper {
    function<void()> run;
    function<int(const Configuration &)> setup;
    function<void()> stop;
    function<string(const string &)> command;
};

class Pool {
public:
    Pool() {
        addModule(make_shared<Logcollector>());
        addModule(make_shared<FIM>());
        addModule(make_shared<Inventory>());
        addModule(make_shared<SCA>());
    }

    template <Module T>
    void addModule(shared_ptr<T> module) {
        auto wrapper = make_shared<ModuleWrapper>(ModuleWrapper{
            .run = [module]() { module->run(); },
            .setup = [module](const Configuration & config) { return module->setup(config); },
            .stop = [module]() { return module->stop(); },
            .command = [module](const string & query) { return module->command(query); }
        });

        modules[module->name()] = wrapper;
    }

    shared_ptr<ModuleWrapper> getModule(const string & name) {
        return modules.at(name);
    }

    void start() {
        for (const auto &[_, module] : modules) {
            module->run();
        }
    }

    void setup(const Configuration & config) {
        for (const auto &[_, module] : modules) {
            module->setup(config);
        }
    }

    void stop() {
        for (const auto &[_, module] : modules) {
            module->stop();
        }
    }

private:
    map<string, shared_ptr<ModuleWrapper>> modules;
};

/******************************************************************************/

int main() {
    Pool pool;
    Configuration config;

    pool.start();
    pool.setup(config);
    cout << endl;

    try {
        auto logcollector = pool.getModule("logcollector");
        logcollector->command("Hello World!");
    } catch (const out_of_range & e) {
        cerr << "!  OOPS: Module not found." << endl;
    }

    cout << endl;
    pool.stop();

    return 0;
}

Conclusion

In conclusion, here is the updated table of pros and cons:

	Inheritance	Wrapper (concepts)
Pros	The agent can transparently receive modules.	Modules are developed independently from the module pool.
Cons	Dependency on the base class. Slight runtime overhead due to vtable usage.	The pool must maintain each module. Slight compilation overhead due to templates, and increased binary size.

Tremendous thanks to @gdiazlo and @Dwordcito for their collaboration in this research.

[jr0me]

Update on implementation restriction no. 4

After investigating the implementation restriction of supporting multiple communications without multithreading, I explored the use of coroutines. Coroutines allow for non-blocking operations by suspending and resuming execution, which can efficiently manage asynchronous tasks. However, achieving true multitasking and supporting multiple concurrent connections may still require an underlying mechanism, such as an event loop or a thread pool, to effectively handle these connections concurrently.

Here's a test using cppcoro (https://github.com/lewissbaker/cppcoro): 726a299 which implements a task pool using coroutines and threads.