Logs not visible in SkyWalking UI after configuring Kafka-fetcher

[abhishekacharya828]

I am using Filebeat to send logs to a Kafka topic. After that, I want to visualize those logs in the SkyWalking server. I have successfully sent logs to the Kafka topic using Filebeat, and I have also configured the Kafka-fetcher in SkyWalking to receive logs from the Kafka topic. However, the logs are not visible in the SkyWalking server UI.

Could you please help me troubleshoot this issue? I would be happy to provide any additional information or logs that might be helpful in resolving this problem.

Thank you in advance for your assistance.

[wu-sheng]

skywalking/oap-server/server-fetcher-plugin/kafka-fetcher-plugin/src/main/java/org/apache/skywalking/oap/server/analyzer/agent/kafka/provider/handler/LogHandler.java

Lines 71 to 87 in 0b9bbb4

	public void handle(final ConsumerRecord<String, Bytes> record) {
	try (HistogramMetrics.Timer ignore = histogram.createTimer()) {
	LogData logData = parseConsumerRecord(record);
	logAnalyzerService.doAnalysis(logData, null);
	} catch (Exception e) {
	errorCounter.inc();
	log.error(e.getMessage(), e);
	}
	}
	protected String getDataFormat() {
	return "protobuf";
	}
	protected LogData parseConsumerRecord(ConsumerRecord<String, Bytes> record) throws Exception {
	return LogData.parseFrom(record.value().get());
	}

According to codes, Kafka fetcher only supports proto objects, I am not sure how you do that with filebeats. I remember only HTTP receiver supports JSON format, which is supported by Filebeat

[abhishekacharya828]

If you need to review any configurations on my local system, please let me know. Additionally, I require assistance with configuring log fetching in SkyWalking. I have followed the official documentation but encountered issues. Could you please advise on the basic configuration for this ?

[wu-sheng]

I explained it, do you follow? I mean it is not supported through your way.

[abhishekacharya828]

Yes, I understand. It seems that my approach is not supported. Could you please provide guidance on the appropriate method to achieve this ?

[wu-sheng]

You could use filebeat HTTP to send JSON.

https://skywalking.apache.org/docs/main/latest/en/api/log-data-protocol/#http-api

[abhishekacharya828]

As per your instructions, logs are sent directly to the SkyWalking port, bypassing Kafka. However, I would like the logs to go through a pipeline like Filebeat => Kafka => SkyWalking. It's worth noting that this is an HTTP API, and Filebeat does not support sending data in this manner.

Filebeat.yml

output section

[wu-sheng]

I said, HTTP endpoint works. Kafka works for logs, just not filebeat. That is filebeat limitation.

[abhishekacharya828]

Can I use fluentd to transfer logs to skywalking ?

[wu-sheng]

No matter what agent you use, most of them are JSON based, and require HTTP endpoint.

If you insist Kafka, we have to extend codes of OAP to add Kafka+JSON raw text support. I am afraid, it is not supported yet.

[abhishekacharya828]

can i directly send logs to skywalking server by using logstash ?

[wu-sheng]

Wasn't I saying clearly about APIs?
We don't care what you are using to collect, you only pick HTTP or Kafka(Protobuf data).