Example setting up SkyWalking Satellite

[andrewgkew]

I realize that the configuration for satellite is very dependent on how you want to use it but I am looking for an example/demo of a standard use case for setting up satellite in k8s please?

Looking at the docs at the configuration being broken down into 3 parts it looks like the most common usage is:

1. Satellite running as a side car to your application
2. The common configurations can stay as is for this example
3. The sharing needs a kafka broker and prometeus instance for client and server
4. I am not 100% sure on the pipeline config if no processing is needed

Do you have a walkthrough that could help me get this setup in k8s? There is an image in dockerhub that I can leverage and then I guess all I need to do is set the correct env vars on the side car

Any help here would be greatly appreciated :)

[wu-sheng]

Satellite 0.1.0 is very far away from out of box. 0.2.0 will be the first version to cover most protocols(trace, metrics, log), the existing 0.1.0 is for log forward only.

@mrproliu will try to initial 0.2.0 after he enhance the document about several typical ways of use.

[wu-sheng]

That is strange. You should have opened debug. Notice, the agent log should be in the agent log folder, rather than service logs.

[andrewgkew]

Looked at logs in agent directory thanks for heads up got these debug logs

DEBUG 2021-09-13 08:21:53:918 grpc-nio-worker-ELG-1-1 org.apache.skywalking.apm.dependencies.io.grpc.netty.NettyClientHandler : [id: 0xe9888942, L:/192.168.68.180:52276 - R:skywalking-oap.skywalking.svc.cluster.local/10.0.49.249:11800] OUTBOUND HEADERS: streamId=164083 headers=GrpcHttp2OutboundHeaders[:authority: skywalking-oap.skywalking.svc.cluster.local:11800, :path: /skywalking.v3.TraceSegmentReportService/collect, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.32.1, agent-version: 8.8.0-SNAPSHOT, grpc-accept-encoding: gzip, grpc-timeout: 29999981u] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false
DEBUG 2021-09-13 08:21:53:918 grpc-nio-worker-ELG-1-1 org.apache.skywalking.apm.dependencies.io.grpc.netty.NettyClientHandler : [id: 0xe9888942, L:/192.168.68.180:52276 - R:skywalking-oap.skywalking.svc.cluster.local/10.0.49.249:11800] OUTBOUND DATA: streamId=164083 padding=0 endStream=true length=290 bytes=000000011d0a3532343730626133383262396134633634396133353763346539353038663633362e34302e313633313532313331333931333439333312353234...
DEBUG 2021-09-13 08:21:53:919 grpc-nio-worker-ELG-1-1 org.apache.skywalking.apm.dependencies.io.grpc.netty.NettyClientHandler : [id: 0xe9888942, L:/192.168.68.180:52276 - R:skywalking-oap.skywalking.svc.cluster.local/10.0.49.249:11800] INBOUND HEADERS: streamId=164083 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc, grpc-encoding: identity, grpc-accept-encoding: gzip, x-envoy-upstream-service-time: 0, date: Mon, 13 Sep 2021 08:21:53 GMT, server: envoy] padding=0 endStream=false

Doesnt look like any errors?

[wu-sheng]

No, this is showing netty(some at least) is working well.

[wu-sheng]

path: /skywalking.v3.TraceSegmentReportService/collect

This shows trace reporting works.

[wu-sheng]

@kezhenxu94 @dmsolr Anything in your mind could make this happens?

@EvanLjp has done local tests, they work well.
I just checked our e2e for logging, the commit id is very closing to the latest.

skywalking/test/e2e/e2e-test/docker/Dockerfile.provider

Line 18 in b20f121

FROM ghcr.io/apache/skywalking-java/jdk-${SW_AGENT_JDK_VERSION}:2b519320559ba94131b098e1258bd5fe41d487e4

[wu-sheng]

found this very interesting project open sourced by Salesforce - a generic framework that can be used for injecting sidecars: https://github.com/salesforce/generic-sidecar-injector

FYI @mrproliu about the Satellite.

[kezhenxu94]

@andrewgkew just pulled your Docker image quadcorps/helloworld-logger and run to find the problem, which is very tricky and I ever saw this in my tests, the answer is that, Spring Boot uses logback by default unless explicitly specified. A quick fix is to exclude all logback dependencies in your project, and you're all set.

I don't have your source codes so my fix was

docker run -it --rm -v $(pwd):/app --entrypoint=sh quadcorps/helloworld-logger -c 'cp logging.jar /app/'
tar -tf logging.jar | egrep 'slf4j|logback' | xargs zip -d logging.jar

I just run the modified logging.jar locally and it works as expected, so you might want to rebuild the Docker image based on it

[andrewgkew]

Ok so where do I go from here?
I application is logging to skywalking OAP but the logs are not showing the the traces?
What do you recommend I do to correct this?

[wu-sheng]

What do you see here? You can't have the link?

[andrewgkew]

@wu-sheng - correct there is no link in the traceid column which is why its not connected to the traces

The logs are generated when I call my API which create the trace. I am producing logs via the log4j2 appender in my spring boot application in the spring boot service

[wu-sheng]

I don't have more idea about what is happening.
You could take a look the application source codes we are using for the demo env. https://github.com/SkyAPMTest/skywalking-live-demo

[kezhenxu94]

@andrewgkew a (not so quick but working) means to verify whether it's configuration error

• add the trace id in the logs
• send the logs simultaneously to both the console (or local file) and SkyWalking OAP
• pick some trace ids from the local logs, and query the trace via the trace id in SkyWalking UI.
• check whether there are logs associated to that trace
  • if no, it's configuration error (or our bug).
  • if yes, you are lucky. b/c there are cases where the logs and traces are not 1-1, so you might be lucky enough to always see the 0-1 or 1-0 relation in the page

[wu-sheng]

@andrewgkew Hope you are doing well on the Kong Summit.

SkyWalking has blogs on website(https://skywalking.apache.org/blog/) and China local Video platform(https://space.bilibili.com/390683219).

I want to ask, do you have video for me to download, and upload to youtube(or link if you already have one) and bilibili(we need to upload manually)

[andrewgkew]

@wu-sheng - thanks Summit went very well thanks.

Kong will upload all the videos (my included) onto youtube in about 2-3 weeks on their channel: https://youtube.com/c/KongInc
Once they are published i will send the link over for you no problem :)

There was an interesting announcement at Summit which may have an impact on SW, Kong Gateway will now support Istio integration: https://konghq.com/solutions/istio-gateway/ I need to look into the specifics before knowing more.

[wu-sheng]

Thanks for update. I keep updating with @moonming and Apache APISIX, they are moving OpenResty ecosystem tonto the top of Istio Mesh.
It would be great to have ALS or similar things from Kong too. We are planning something for APISIX.