Skip to content

javascript processor: replace global paths with per-beat paths #47870

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

javascript processor: replace global paths with per-beat paths #47870

wants to merge 4 commits into from
+189 −40

Conversation

@orestisfl
Copy link
Contributor

@orestisfl orestisfl commented Dec 2, 2025 •
edited
Loading

Proposed commit message

Changes:

  • Add SetPaths(path *paths.Path) method to jsProcessor that accepts a per-beat paths configuration
  • Defer file-based source initialization until SetPaths is called
  • For inline sources, initialization still happens immediately in NewFromConfig
  • loadSources now takes a *paths.Path parameter and uses pathConfig.Resolve() instead of the global paths.Resolve()

Fixes #46988

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
  • I have added an entry in ./changelog/fragments using the changelog tool.

How to test this PR locally

Run filebeat with:

path.config: myconfig/

filebeat.inputs:
  - type: filestream
    id: input-a
    paths:
      - /tmp/logs/a.log

    processors:
      - script:
          lang: javascript
          file: test_processor.js
          tag: test-js-processor

output.console:
  enabled: true

myconfig/test_processor.js:

function process(event) {
    event.Put("js_processor.processed", true);
    event.Put("js_processor.timestamp", new Date().toISOString());

    var msg = event.Get("message");
    if (msg) {
        event.Put("message_upper", msg.toUpperCase());
    }

    return event;
}

write some logs:

yes 'some log' | head -n 10000 > /tmp/logs/a.log

See output:

{
  "@timestamp": "2025-12-02T18:09:36.488Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "9.3.0"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "log": {
    "offset": 89586,
    "file": {
      "path": "/tmp/logs/a.log",
      "device_id": "38",
      "inode": "73159",
      "fingerprint": "83016ba24a8d31ccb16d2230eabcb1f043fa4c65914339eb954619b5c13fd55a"
    }
  },
  "message": "some log",
  "input": {
    "type": "filestream"
  },
  "js_processor": {
    "processed": true,
    "timestamp": "2025-12-02T18:09:36.488Z"
  },
  "message_upper": "SOME LOG",
  "host": {
    "name": "laptop"
  },
  "agent": {
    "version": "9.3.0",
    "ephemeral_id": "f6bb5c64-d2d1-4dad-a0e1-bbc553b617ec",
    "id": "40aef7d0-efcb-4613-a1df-d9bc42ff36b9",
    "name": "laptop",
    "type": "filebeat"
  }
}

Related issues

@orestisfl
javascript processor: replace global paths with per-beat paths
4133bf2 
Changes:
- Add SetPaths(path *paths.Path) method to jsProcessor that accepts
  a per-beat paths configuration
- Defer file-based source initialization until SetPaths is called
- For inline sources, initialization still happens immediately in
  NewFromConfig
- loadSources now takes a *paths.Path parameter and uses
  pathConfig.Resolve() instead of the global paths.Resolve()

Fixes elastic#46988
@orestisfl orestisfl requested a review from leehinman December 2, 2025 17:53
@orestisfl orestisfl self-assigned this Dec 2, 2025
@orestisfl orestisfl requested a review from a team as a code owner December 2, 2025 17:53
@orestisfl orestisfl added enhancement backport-skip Skip notification from the automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team skip-changelog labels Dec 2, 2025
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 2, 2025
@elasticmachine
Copy link
Contributor

elasticmachine commented Dec 2, 2025

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 2, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2025

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@orestisfl orestisfl marked this pull request as draft December 2, 2025 18:13
@orestisfl orestisfl marked this pull request as ready for review December 2, 2025 18:23
@orestisfl orestisfl requested a review from a team as a code owner December 2, 2025 18:23
@pierrehilbert
Copy link
Contributor

pierrehilbert commented Dec 3, 2025

@qcorporation we will need someone from your team to review here please.

jrmolin
jrmolin reviewed Dec 3, 2025
View reviewed changes
Copy link
Contributor

@jrmolin jrmolin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't see how the change encapsulates the original implementation

libbeat/processors/script/javascript/javascript_test.go
assert.ErrorContains(t, err, "SetPaths must be called")
})

t.Run("after SetPaths on inline source", func(t *testing.T) {
Copy link
Contributor

@jrmolin jrmolin Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably should also test that it works without SetPaths on an inline source

Copy link
Contributor Author

@orestisfl orestisfl Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested in with inline source

libbeat/processors/script/javascript/javascript_test.go Outdated
assert.Equal(t, true, v)
})

t.Run("without SetPaths fails", func(t *testing.T) {
Copy link
Contributor

@jrmolin jrmolin Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you test that it works with a SetPaths call?

Copy link
Contributor Author

@orestisfl orestisfl Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed a commit that covers it in the with file test

libbeat/processors/script/javascript/javascript.go
var err error

switch {
case p.File != "":
Copy link
Contributor

@jrmolin jrmolin Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't see where p.File or p.Files would be set -- are they in the config? i don't know that this is getting executed anywhere.

Copy link
Contributor Author

@orestisfl orestisfl Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, NewFromConfig which sets Config which is embedded in:

type jsProcessor struct {
	Config

@orestisfl
Copy link
Contributor Author

orestisfl commented Dec 3, 2025

i don't see how the change encapsulates the original implementation

@jrmolin do you mean if the change intends to be backwards compatible from an API perspective? If yes, we settled on not allowing using with the default global paths: #47353 (comment) so the SetPaths call is mandatory.

@orestisfl orestisfl requested a review from jrmolin December 3, 2025 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leehinman leehinman Awaiting requested review from leehinman

@khushijain21 khushijain21 Awaiting requested review from khushijain21 khushijain21 is a code owner automatically assigned from elastic/elastic-agent-data-plane

@jrmolin jrmolin Awaiting requested review from jrmolin

At least 1 approving review is required to merge this pull request.

Assignees

@orestisfl orestisfl

Labels

backport-skip Skip notification from the automated backport with mergify enhancement skip-changelog Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[beatreceiver] replace global paths in javascript processor

4 participants

@orestisfl @elasticmachine @pierrehilbert @jrmolin