add file retry events and adjust lock structure to fix eBPF file data loss #2292

xiongyunn · 2025-07-04T03:05:54Z

No description provided.

Merge file_security into local

* feat: add instance labels into host monitor metrics

* feat: support send multivalue metrics to sls * fix * fix * fix * fix * fix

…g reading. (alibaba#2246) * fix and add ut * polish * polish

* add e2e framework code for eBPF process collection * e2e support ssh to host by private key * fix: formatting errors

* feat: provide common cache for system information

* trim meaningless \0 when reading file * remove redundant code

… function. (alibaba#2256) * fix container path format * fix lint * fix commentf

* Disable process security local container meta * add btf search path * set logger for any bpf program * stop process manager if not inited successfully

* upgrade go version from 1.19.10 to 1.23.10 * update dev image * fix license * remove goc * update build script * update go lint * update go lint * fix lint * fix e2e * speedup ci * fix static check * fix go version

…" (alibaba#2260) * Revert "Feat: upgrade go version from 1.19.10 to 1.23.10 (alibaba#2254)" This reverts commit f1712da. * update workflow

* feat: windows build * feat: add windows workflows * fix: windows build script --------- Co-authored-by: linrunqi08 <[email protected]> Co-authored-by: Takuka0311 <[email protected]>

Fix crash caused by pointer moved to nullptr when camel concurrent queue is full Use eBPF Server's polling thread to run process cache management Use eBPF Server's handling thread to run plugin's aggregation and send Refine eBPF Server's start and stop routine

* fix * fix lint and ut memory leaks * fix lint and ut * fix memory leaks * fix GCC diagnostic push * polish * add self-monitor benchmark * remove test branch * Revert "remove test branch" This reverts commit f31a3bb. * Revert "add self-monitor benchmark" This reverts commit 87015b3.

* feat: windows build * feat: add windows workflows * fix: windows build script --------- Co-authored-by: linrunqi08 <[email protected]> Co-authored-by: Takuka0311 <[email protected]>

merge eBPF thread structure modifications

merge new commit

CLAassistant · 2025-07-15T08:07:22Z

All committers have signed the CLA.

yyuuttaaoo · 2025-07-16T13:48:39Z

core/ebpf/Config.cpp

@@ -14,6 +14,7 @@

 #include "ebpf/Config.h"

+#include <set>


这个没用到

yyuuttaaoo · 2025-07-16T13:52:13Z

core/ebpf/Config.cpp

+        size_t originalSize = thisFileFilter.mFilePathList.size();
+        std::unordered_set<std::string> uniquePaths;
+        std::vector<std::string> deduplicatedPaths;
+        deduplicatedPaths.reserve(originalSize);


这个去重功能有UT吗

在InputFileSecurityUnittest里面有一个

yyuuttaaoo · 2025-07-16T13:58:24Z

core/ebpf/EBPFServer.cpp

        default:
            LOG_ERROR(sLogger, ("unknown plugin type", int(type)));
            return false;
    }

    if (pluginMgr->Init(options) != 0) {
+        LOG_ERROR(sLogger, ("plugin manager init failed", ""));
+        if ((type == PluginType::NETWORK_SECURITY || type == PluginType::PROCESS_SECURITY


这里条件要个上面Init的地方保持一致。最好就上面判断一下后写一个flag，都通过这个flag决定要不要init和stop processcachemanager

yyuuttaaoo · 2025-07-17T03:59:16Z

core/ebpf/EBPFServer.cpp

+
+        int processCacheEvents = mProcessCacheManager->PollPerfBuffers();
+
+        int currentMaxWaitTime = kDefaultMaxWaitTimeMS;


这好像不怎么符合预期，应该是这样的逻辑

maxWaitTimeMs=kDefaultMaxWaitTimeMs
endtime = now()

starttime = endtime
pollperfbuffer(maxWaitTimeMs)
endtime = now()
maxWaitTimeMs -= (endtime - starttime)
if (maxWaitTimeMs < 0) maxWaitTimeMs = 1

starttime = endtime
……

查了一下接下来确实可以改为嵌套epoll的逻辑

yyuuttaaoo · 2025-07-17T04:01:10Z

core/ebpf/EBPFServer.h

@@ -60,6 +66,8 @@ struct PluginState {
    // (PollPerfBuffers/HandlerEvents/GetAllProjects), allowing them to safely interleave.
    mutable std::atomic_bool mValid;
    mutable std::shared_mutex mMtx;
+    std::atomic<LifecycleState> mLifecycleState{LifecycleState::STOPPED};


通过框架保证type变更后stop(true)是不是这些逻辑就可以不要了

yyuuttaaoo · 2025-07-17T04:12:56Z

core/ebpf/driver/eBPFDriver.cpp

+    for (auto* pb : pbs) {
+        auto* perfbuffer = static_cast<perf_buffer*>(pb);
+        if (perfbuffer) {
+            perf_buffer__free(perfbuffer);


yyuuttaaoo · 2025-07-17T06:59:42Z

core/ebpf/plugin/RetryableEvent.h

@@ -59,7 +59,7 @@ class RetryableEvent {
     * @brief Checks if the event can be retried.
     * @return true if there are retry attempts left, false otherwise.
     */
-    [[nodiscard]] bool CanRetry() const { return mRetryLeft > 0; }
+    [[nodiscard]] virtual bool CanRetry() const { return mRetryLeft > 0; }


这个不应该需要override

yyuuttaaoo · 2025-07-17T07:11:27Z

core/ebpf/Config.cpp

@@ -228,7 +229,29 @@ void InitSecurityFileFilter(const Json::Value& config,
    } else if (!GetOptionalListFilterParam<std::string>(
                   config, "FilePathFilter", thisFileFilter.mFilePathList, errorMsg)) {
        // FilePathFilter has element of wrong type
+    } else {
+        // FilePathFilter succeeded, deduplication
+        size_t originalSize = thisFileFilter.mFilePathList.size();


这里验证一下配置，应该有个filter数量上限或者总长度上限，与ebpf程序上限对应

yyuuttaaoo and others added 26 commits April 18, 2025 10:51

[WIP]file security

d6b5fc1

Merge remote-tracking branch 'xunfei/dev/file_security'

5abc5b1

Merge file_security into local

add eBPF functionality to collect file read and write operations

a86f2fd

feat: add instance labels into host monitor metrics (alibaba#2226)

1f84a54

* feat: add instance labels into host monitor metrics

feat: support send multivalue metrics to sls (alibaba#2233)

6f7547a

* feat: support send multivalue metrics to sls * fix * fix * fix * fix * fix

Fix: JsonLogFileReader may cause a crash due to buffer overflow durin…

5133712

…g reading. (alibaba#2246) * fix and add ut * polish * polish

add e2e framework code for eBPF process collection (alibaba#2250)

600a48a

* add e2e framework code for eBPF process collection * e2e support ssh to host by private key * fix: formatting errors

feat: provide common cache for system information (alibaba#2212)

38414b3

* feat: provide common cache for system information

trim meaningless \0 when reading file (alibaba#2251)

3f6aca0

* trim meaningless \0 when reading file * remove redundant code

Unify the file path formatting in container meta information into one…

9cfb983

… function. (alibaba#2256) * fix container path format * fix lint * fix commentf

Disable process security local container meta (alibaba#2255)

a49f15c

* Disable process security local container meta * add btf search path * set logger for any bpf program * stop process manager if not inited successfully

Feat: upgrade go version from 1.19.10 to 1.23.10 (alibaba#2254)

d944ead

* upgrade go version from 1.19.10 to 1.23.10 * update dev image * fix license * remove goc * update build script * update go lint * update go lint * fix lint * fix e2e * speedup ci * fix static check * fix go version

Revert "Feat: upgrade go version from 1.19.10 to 1.23.10 (alibaba#2254)…

ead3545

…" (alibaba#2260) * Revert "Feat: upgrade go version from 1.19.10 to 1.23.10 (alibaba#2254)" This reverts commit f1712da. * update workflow

feat: add system collector (alibaba#2231)

80a47ba

feat: loongcollector support windows build (alibaba#2082)

5430d93

* feat: windows build * feat: add windows workflows * fix: windows build script --------- Co-authored-by: linrunqi08 <[email protected]> Co-authored-by: Takuka0311 <[email protected]>

fix docker inspect timeout (alibaba#2269)

57051d9

[WIP]file security

1af26fa

add eBPF functionality to collect file read and write operations

4785ba0

feat: loongcollector support windows build (alibaba#2082)

7fb02bf

* feat: windows build * feat: add windows workflows * fix: windows build script --------- Co-authored-by: linrunqi08 <[email protected]> Co-authored-by: Takuka0311 <[email protected]>

Merge branch 'alibaba-main'

54889e9

merge eBPF thread structure modifications

fix data loss issues

abb8679

Merge pull request #2 from alibaba/main

a779766

merge new commit

fix plugin type update issue

3bc6a2d

improve eBPF file performance

609ec7a

xiongyunn closed this Jul 11, 2025

xiongyunn reopened this Jul 11, 2025

xiongyunn changed the title ~~[WIP] file security~~ add file retry events and adjust lock structure to fix eBPF file data loss Jul 11, 2025

add deduplication logic for the whitelist

52557bc

xiongyunn force-pushed the main branch from ec6f95b to 52557bc Compare July 15, 2025 08:39

xiongyunn added 3 commits July 15, 2025 08:42

fix the unit tests for ebpf file

04644fe

dynamically adjust the MaxWaitTime of pollperfbuffer

eac7f56

adjust the data reporting format

312c1e1

yyuuttaaoo reviewed Jul 17, 2025

View reviewed changes

fix issue and add ut

f3acd1b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

add file retry events and adjust lock structure to fix eBPF file data loss #2292

add file retry events and adjust lock structure to fix eBPF file data loss #2292

xiongyunn commented Jul 4, 2025

Uh oh!

CLAassistant commented Jul 15, 2025 •

edited

Loading

Uh oh!

yyuuttaaoo Jul 16, 2025

Uh oh!

yyuuttaaoo Jul 16, 2025

Uh oh!

xiongyunn Jul 17, 2025

Uh oh!

yyuuttaaoo Jul 16, 2025

Uh oh!

yyuuttaaoo Jul 17, 2025

Uh oh!

yyuuttaaoo Jul 17, 2025

Uh oh!

yyuuttaaoo Jul 17, 2025

Uh oh!

yyuuttaaoo Jul 17, 2025

Uh oh!

yyuuttaaoo Jul 17, 2025

Uh oh!

yyuuttaaoo Jul 17, 2025

Uh oh!

Uh oh!


		int processCacheEvents = mProcessCacheManager->PollPerfBuffers();

		int currentMaxWaitTime = kDefaultMaxWaitTimeMS;

add file retry events and adjust lock structure to fix eBPF file data loss #2292

Are you sure you want to change the base?

add file retry events and adjust lock structure to fix eBPF file data loss #2292

Conversation

xiongyunn commented Jul 4, 2025

Uh oh!

CLAassistant commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CLAassistant commented Jul 15, 2025 •

edited

Loading