[Proposal] Introduce language-specific profiling in SkyWalking

[lujiajing1126]

Hi the community,

I would like to submit a new proposal to introduce continuous profiling to the SkyWalking ecosystems,

A draft can be found here,
https://docs.google.com/document/d/1Huf1sWQdikMoJxxCRid-P7tqF0mGIS7zMm0DwumRiQM/edit?usp=sharing

More details would be added if it is accepted.

[wu-sheng]

@lujiajing1126 Thanks for proposing this. Some of us discussed what is continues profiling, and the general idea is to activate any existing profiling mechanism we have, and do the profiling when necessary.

In your proposal, it seens more like introducing the async-profiling, rather than continues profiling.

Could you be more clear about the scope of your proposal?

FYI @mrproliu

[lujiajing1126]

Yes. I think the language-specific profiling is a better name for this proposal. eBPF is already an implementation.

"Continuous" means we always turn on the switch which could be a challenge for the storage and cause performance issue. But we could introduce async-profiler-based profiling and pprof-based ones first.

[wu-sheng]

Sure. Let's forcus on on-demand profiling, including those too.

Then, we could think about the continues profiling. Notice, from what I saw, continues profiling never means keep profiling as always, because no matter how low impact, profiling has much more overhead than metrics and tracing, which is why those are still the mainstream.
Continues profiling is all profiling tools could be activated automatically when some thresholds(most likely metrics or specific events) reached.

[wu-sheng]

Also, keeping profiling offers low value for observability. If a prod env needs to be watched/profilied 7*24, it is not prod ready. Profiling is a kind of diagnosing for specific cases.
APM wouldn't be the best debug tool for dev/test stage, and we should not try to do so.

[mrproliu]

I agree with Sheng's viewpoint that "on-demand" profiling may be more appropriate. I believe that continuous profiling can be achieved by periodically collecting certain metrics or events, and activating existing profiling tools (including in-process or out-of-process mode) when certain trigger conditions are met.
Additionally, I suggest considering adding tools such as async-profiler which can dynamically attach to the target process in out-of-process mode, to skywalking-rover. This would have the advantages of allowing users to complete performance analysis without updating the agent, and improving the shortcomings of OnCPU Profiling for Java programs. However, I am not sure if it supports other modes of profiling, such as OffCPU profiling.

[lujiajing1126]

I completely agree with both of you for the "on-demand" concept. The marginal utility would decrease with increasing profiling time. And I like the idea of metrics-threshold-triggered profiling, but it seems an advanced function.

Additionally, I suggest considering adding tools such as async-profiler which can dynamically attach to the target process in out-of-process mode, to skywalking-rover.
I'm not familiar with the skywalking-rover. Integrating with the Java-Agent is straight forward for me. But I guess "dynamically attachment" is possible according to what "Alibaba/Arthas" has done.

I will discuss further with @mrproliu with the current eBPF profiling and the skywalking-rover project. In the meanwhile I will complete the proposal. Maybe we can organize an online meeting after the Spring Festival.

And thanks @Superskyyy for the insterst to provide Python integration.