Implement method call stack tracing #150

sohyun-ku · 2025-04-21T08:19:38Z

resolved: implement method call stack #109
Call Stack Tracing with ConcurrentHashMap + Deque
- To manage call stacks per thread, we used the structure ConcurrentHashMap<Long, ArrayDeque<String>>.
- Each thread maintains its own call stack (Deque), and method entries and exits are tracked via push and pop operations respectively.
- The Deque is used as a stack to accurately capture the order and depth of method calls.
Caller-Callee Based Lightweight Call Structure
- Instead of storing the entire method call stack, each call holds a reference to its direct caller only.
- This design retains the hierarchical relationship between method calls while significantly reducing memory overhead by avoiding redundant stack information.
- It allows for a clear understanding of “who called whom” at the direct (1-depth) level,
  and can be effectively used for incremental exploration or future enhancements such as call relationship visualization.

Benchmark Results (ns/op)

The benchmarking tests were conducted on a local machine with an Apple M1 Pro, 32GB RAM. Please note that results may vary depending on the environment.
Since enabling CallStackTraceMode introduces some overhead, the feature is implemented as optional, allowing it to be turned on only when detailed call tracing is needed.

Java version	CallStackTraceMode False	CallStackTraceMode True
Java 8	14.056 ± 0.235	34.004 ± 0.403
Java 11	12.933 ± 0.247	27.244 ± 0.344
Java 17	12.831 ± 0.340	26.625 ± 0.248
Java21	12.502 ± 0.104	26.408 ± 3.061

Snapshot: Caller-Callee Relationship Example

github-actions · 2025-04-21T08:25:56Z

Scavenger Test Results

178 files 178 suites 3m 28s ⏱️
304 tests 297 ✅ 7 💤 0 ❌
326 runs 319 ✅ 7 💤 0 ❌

Results for commit cf1d43d.

♻️ This comment has been updated with latest results.

taeyeon-Kim

Sorry for the late review. I only checked the agent and api modules first.

Is it possible to see the call stack tree maybe?
Also, could you please change the vitess configuration file.

...agent-java/src/main/java/com/navercorp/scavenger/javaagent/collecting/CallStackRegistry.java

...-agent-java/src/main/java/com/navercorp/scavenger/javaagent/collecting/CallStackTracker.java

taeyeon-Kim · 2025-05-12T09:24:00Z

...ger-agent-java/src/test/java/com/navercorp/scavenger/javaagent/scheduling/SchedulerTest.java

@@ -81,7 +88,9 @@ public class SchedulerTest {
    public void setUp() throws Exception {
        lenient().when(codeBaseScannerMock.scan())
            .thenReturn(new CodeBase(List.of(Method.createTestMethod()), "fingerprint"));
-        sut = new Scheduler(invocationRegistry, new Config(new Properties()), publisher, codeBaseScannerMock);
+        Properties properties = new Properties();
+        properties.setProperty("callStackTraceMode", "true");


To make it more clear, it also needs to be tested when callStackTraceMode is false

scavenger-agent-java/src/main/java/com/navercorp/scavenger/javaagent/scheduling/Scheduler.java

taeyeon-Kim · 2025-05-12T09:47:35Z

scavenger-api/src/main/kotlin/com/navercorp/scavenger/repository/sql/CallStackSql.kt

+        invokedAtMillis: Long?,
+    ): String =
+        """
+        SELECT


Can you check query performance?

EXPLAIN FORMAT=VITESS {query}

Sorry for the late reply :(
Here is the Vitess query plan

+----------+-------------------+----------+-------------+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | operator | variant | keyspace | destination | tabletType | query | +----------+-------------------+----------+-------------+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Route | SelectEqualUnique | main | | UNKNOWN | select caller_methods.signature as callerSignature from call_stacks join methods as callee_methods on call_stacks.customerId = callee_methods.customerId and call_stacks.signatureHash = callee_methods.signatureHash join methods as caller_methods on call_stacks.customerId = caller_methods.customerId and call_stacks.callerSignatureHash = caller_methods.signatureHash where callee_methods.signature = '${PATH}' and call_stacks.customerId = 11000002003 and call_stacks.applicationId in (11003142347) and call_stacks.environmentId in (11003142346) and (1751941260000 is null or call_stacks.invokedAtMillis >= 1751941260000) | +----------+-------------------+----------+-------------+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec)

scavenger-collector/src/main/kotlin/com/navercorp/scavenger/controller/GrpcAgentController.kt

scavenger-api/src/main/kotlin/com/navercorp/scavenger/repository/sql/CallStackSql.kt

taeyeon-Kim · 2025-05-13T07:42:48Z

scavenger-collector/src/main/kotlin/com/navercorp/scavenger/repository/MethodDao.kt

@@ -59,7 +59,7 @@ class MethodDao(
            return 0
        }
        return update(
-            sql.deleteAllMethodsAndInvocations(),
+            sql.deleteAllMethods(),


sohyun-ku · 2025-06-04T08:32:55Z

@taeyeon-Kim

Is it possible to see the call stack tree maybe?

In the current PR, the implementation is based on a one-way caller-callee relationship. However, I expect that in future feature expansions, it will be possible to construct the full call tree by recursively tracing the call path in reverse from a specific callee.
I considered various ways to model the call structure, but ultimately chose this simple caller-callee approach for its simplicity and ease of maintenance.
If you see any limitations in the current structure or believe another approach might be more appropriate, I’d appreciate your feedback.

taeyeon-Kim · 2025-06-18T05:19:02Z

@sohyun-ku This is a great idea, and for an early version, this is great!!
Can I ask you to check one thing, if it is possible to extend this, for example, to make it possible to draw a tree from Caller to Root Caller.

sohyun-ku · 2025-06-20T07:32:25Z

@taeyeon-Kim

I think it is technically possible to build a tree from a specific callee up to the root
(e.g., by traversing the reversed graph recursively to explore all possible call paths).

However, the stored call data represents multiple call records across the application rather than a single call flow.
Therefore, the constructed tree resembles a combination of all possible call relationships rather than an actual call stack.

Additionally, since the call graph may contain cycles and paths can get deep,
it’s important to apply constraints such as a visited set to avoid duplicates and depth limits to prevent excessive recursive traversal.

In summary, while extending the tree up to the root is feasible, please note that this does not exactly reproduce a single actual call flow that triggered the callee.

Due to the necessity of minimizing the performance impact on the agent,
these constraints inevitably arose to reduce performance and memory overhead during tree construction.

taeyeon-Kim

🔥 👍 👍 👍 👍

taeyeon-Kim · 2025-06-24T08:31:51Z

@taeyeon-Kim

I think it is technically possible to build a tree from a specific callee up to the root (e.g., by traversing the reversed graph recursively to explore all possible call paths).

However, the stored call data represents multiple call records across the application rather than a single call flow. Therefore, the constructed tree resembles a combination of all possible call relationships rather than an actual call stack.

Additionally, since the call graph may contain cycles and paths can get deep, it’s important to apply constraints such as a visited set to avoid duplicates and depth limits to prevent excessive recursive traversal.

In summary, while extending the tree up to the root is feasible, please note that this does not exactly reproduce a single actual call flow that triggered the callee.

Due to the necessity of minimizing the performance impact on the agent, these constraints inevitably arose to reduce performance and memory overhead during tree construction.

@sohyun-ku yes I understand, thanks for the clarification. 😄

sohyun-ku added 10 commits April 17, 2025 17:09

Add proto

4d70bed

Implement to track call stack

5a190b4

Add TC

116feea

Update BenchmarkTest

a7acd0e

Implement to import call stack

5977286

Add call stack GC

105aff2

Clean up code

736dfbc

Add exception handling

f8c77fb

Add GC Test

8e21507

Implement API & FE

0609fd1

sohyun-ku self-assigned this Apr 21, 2025

sohyun-ku requested review from junoyoon and a team as code owners April 21, 2025 08:19

Update installation.md

333ebdf

taeyeon-Kim reviewed May 12, 2025

View reviewed changes

taeyeon-Kim reviewed May 13, 2025

View reviewed changes

sohyun-ku added 2 commits June 4, 2025 16:31

Clean up code

c54cc38

Fix scheduler TC

42e0e08

Add vitess configuation

cf1d43d

taeyeon-Kim approved these changes Jun 24, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement method call stack tracing #150

Implement method call stack tracing #150

Uh oh!

sohyun-ku commented Apr 21, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Apr 21, 2025 •

edited

Loading

Uh oh!

taeyeon-Kim left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

taeyeon-Kim May 12, 2025

Uh oh!

Uh oh!

taeyeon-Kim May 12, 2025

Uh oh!

sohyun-ku Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!

taeyeon-Kim May 13, 2025

Uh oh!

sohyun-ku commented Jun 4, 2025

Uh oh!

taeyeon-Kim commented Jun 18, 2025

Uh oh!

sohyun-ku commented Jun 20, 2025

Uh oh!

taeyeon-Kim left a comment

Uh oh!

taeyeon-Kim commented Jun 24, 2025

Uh oh!

Uh oh!

Implement method call stack tracing #150

Are you sure you want to change the base?

Implement method call stack tracing #150

Uh oh!

Conversation

sohyun-ku commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Scavenger Test Results

Uh oh!

taeyeon-Kim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

taeyeon-Kim May 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

taeyeon-Kim May 12, 2025

Choose a reason for hiding this comment

Uh oh!

sohyun-ku Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

taeyeon-Kim May 13, 2025

Choose a reason for hiding this comment

Uh oh!

sohyun-ku commented Jun 4, 2025

Uh oh!

taeyeon-Kim commented Jun 18, 2025

Uh oh!

sohyun-ku commented Jun 20, 2025

Uh oh!

taeyeon-Kim left a comment

Choose a reason for hiding this comment

Uh oh!

taeyeon-Kim commented Jun 24, 2025

Uh oh!

Uh oh!

sohyun-ku commented Apr 21, 2025 •

edited

Loading

github-actions bot commented Apr 21, 2025 •

edited

Loading