Skip to content

Commit f6ed702

Browse files
wu-shengsongzhendong
authored and
songzhendong
committed
Add a FAQ about other storage options. (#12059)
1 parent 3e39dea commit f6ed702

File tree

4 files changed

+44
-3
lines changed

4 files changed

+44
-3
lines changed

Diff for: docs/en/FAQ/README.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22
These are known and frequently asked questions about SkyWalking. We welcome you to contribute here.
33

44
## Design
5-
* [Why doesn't SkyWalking involve MQ in its architecture?](why_mq_not_involved.md)
5+
* [Why does SkyWalking use RPC(gRPC and RESTful) rather than MQ as transport layer by default?](why_mq_not_involved.md)
6+
* [Why is Clickhouse or Loki or xxx not supported as a storage option?](why-clickhouse-not-supported.md)
67

78
## Compiling
89
* [Protoc plugin fails in maven build](Protoc-Plugin-Fails-When-Build.md)

Diff for: docs/en/FAQ/why-clickhouse-not-supported.md

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Why is Clickhouse or Loki or xxx not supported as a storage option?
2+
3+
## Background
4+
5+
In the past several years, community users have asked why Clickhouse, Loki, or some other storage is not supported in the upstream. We have repeated the answer many times, but it is still happening, at here, I would like to write down the summary to help people understand more
6+
7+
## Previous Discussions
8+
All the following issues were about discussing new storage extension topics.
9+
- Loki as storage
10+
- https://github.com/apache/skywalking/discussions/9836
11+
- ClickHouse
12+
- https://github.com/apache/skywalking/issues/11924
13+
- https://github.com/apache/skywalking/discussions/9011
14+
- Vertica
15+
- https://github.com/apache/skywalking/discussions/8817
16+
17+
Generally, all those asking are about adding a new kind of storage.
18+
19+
## Why they don't exist ?
20+
First of all, `WHY` is not a suitable question. SkyWalking is a volunteer-driven community, the volunteers build this project including bug fixes, maintenance work, and new features from their personal and employer interests. What you saw about the current status is the combination of all those interests rather than responsibilities.
21+
So, in SkyWalking, anything you saw existing is/was someone's interest and contributed to upstream.
22+
23+
This logic is the same as this question, SkyWalking active maintainers are focusing on JDBC(MySQL and PostgreSQL ecosystem) Database and Elasticsearch for existing users, and moving forward on BanyanDB as the native one. We for now don't have people interested in ClickHouse or any other database. That is why they are not there.
24+
25+
## How could add one?
26+
To add a new feature, including a new storage plugin, you should go through [SWIP - SkyWalking Improvement Proposal](https://skywalking.apache.org/docs/main/next/en/swip/readme/) workflow, and have a full discussion with the maintenance team.
27+
SkyWalking has a pluggable storage system, so, ideally new storage option is possible to implement a new provider for the storage module. Meanwhile, in practice, as storage implementation should be in high performance and well optimized, considering our experiences with JDBC and Elasticsearch implementations, some flags and annotations may need to be added in the kernel level and data model declarations.
28+
29+
Furthermore, as current maintainers are not a fun of Clickhouse or others(otherwise, you should have seen those implementations), they are not going to be involved in the code implementations and they don't know much more from a general perspective about which kind of implementation in that specific database will have a better behavior and performance. So, if you want to propose this to upstream, you should be very experienced in that database, and have enough scale and environments to provide solid benchmark.
30+
31+
## What happens next if the new implementation gets accepted/merged/released?
32+
Who proposed this new implementation(such as clickhouse storage), has to take the responsibilities of the maintenance. The maintenance means they need to
33+
1. Join storage relative discussion to make sure SkyWalking can move forward on a kernel-level optimization without being blocked by these specific storage options.
34+
2. Respond to this storage relative questions, bugs, CVEs, and performance issues.
35+
3. Make the implementation performance match the expectation of the original proposal. Such as, about clickhouse, people are talking about how they are faster and have higher efficiency than Elasticsearch for large-scale deployments. Then we should always be able to see it has better benchmark and product side practice.
36+
37+
Even if the storage gets accepted/merged/released, but **no one can't take the above responsibilities** or **the community doesn't receive the feedback and questions about those storages**, SkyWalking PMC(Project Management Committee) will start the process to remove the implementations. This happened before for Apache IoTDB and InfluxDB storage options. Here is the last vote about this,
38+
- https://github.com/apache/skywalking/discussions/9059

Diff for: docs/en/FAQ/why_mq_not_involved.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Why doesn't SkyWalking involve MQ in its architecture?
1+
# Why does SkyWalking use RPC(gRPC and RESTful) rather than MQ as transport layer by default?
22
This is often asked by those who are first introduced to SkyWalking. Many believe that MQ should have better performance and should be able to support higher throughput, like the following:
33

44
<img src="MQ-involved-architecture.png"/>
@@ -23,4 +23,4 @@ Even though MQ transport is not recommended from the production perspective, Sky
2323
`kafka-reporter` and `kafka-fetcher` for this feature since 8.1.0.
2424

2525
### How about MQ metrics data exporter?
26-
The answer is that the MQ metrics data exporter is already readily available. The exporter module with gRPC default mechanism is there, and you can easily provide a new implementor of this module.
26+
Log and trace exporters are using MQ as transport channel. And metrics exporter uses gRPC, as considering the scale.

Diff for: docs/en/changes/changes.md

+2
Original file line numberDiff line numberDiff line change
@@ -129,5 +129,7 @@
129129
* Add `SWIP-5 Support ClickHouse Monitoring`.
130130
* Remove `OpenTelemetry Exporter` support from meter doc, as this has been flagged as unmaintained on OTEL upstream.
131131
* Add doc of one-line quick start script for different storage types.
132+
* Add FAQ for `Why is Clickhouse or Loki or xxx not supported as a storage option?`.
133+
* Add Airflow monitoring.
132134

133135
All issues and pull requests are [here](https://github.com/apache/skywalking/milestone/202?closed=1)

0 commit comments

Comments
 (0)