OAP Pods are keep failing with Error Starup Probe Faile

[Jatin3708]

Hi Team,

I am trying to deploy skywalking using terrafrom helm release, which is getting failed.

While further checking I found that pods for the OAP are not into running state.

$ kubectl get pods -n skywalking
NAME READY STATUS RESTARTS AGE
skywalking-postgresql-0 1/1 Running 0 11h
skywalking-skywalking-helm-oap-688bbd5dbb-8jfz5 0/1 CrashLoopBackOff 279 (2m16s ago) 18h
skywalking-skywalking-helm-oap-688bbd5dbb-kw7w2 0/1 CrashLoopBackOff 173 (4m40s ago) 11h
skywalking-skywalking-helm-ui-6bb48c8f49-gz7mh 1/1 Running 0 11h

While further describing oap pods found that Startup probe is failied.

Warning Unhealthy 8m21s (x2514 over 18h) kubelet Startup probe failed: dial tcp 100.64.40.194:12800: connect: connection refused

Further investigation done to check logs for the pods and found error that :

2024-12-13 05:05:06,081 org.apache.skywalking.oap.server.core.storage.model.ModelInstaller 43 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting...
retry 3s later.
2024-12-13 05:05:09,082 org.apache.skywalking.oap.server.core.storage.model.ModelInstaller 43 [main] INFO [] - table: alarm_record does not exist. OAP is running in 'no-init' mode, waiting...
retry 3s later.

How to get rid of this error?

Below are setting used :

set {
name = "oap.image.tag"
value = "10.0.0 "
}
set {
name = "oap.storageType"
value = "postgresql"
}
set {
name = "ui.image.tag"
value = "10.0.0"
}
set {
name = "elasticsearch.enabled"
value = "false"
}
set {
name = "postgresql.enabled"
value = "true"
}

[wu-sheng]

You seem to erase the storage and don't rerun the init job.

[Jatin3708]

Hi @wu-sheng : Let me try with RDS instance now.

In helm chart pg address and other all configuration are setup by chart, I have just enabled pgsql and disabled elastic cluster, no other manual stuff are done. So ideally all configuration should be fine.

[wu-sheng]

I usually don't run pg part, sorry. Usually, elastic and banyandb work for me.

[Jatin3708]

@wu-sheng :

I tried with all 3 DB type now, even BanyanDB and Elasticsearch with helm-chart release 4.7.0, all are failing with same error !!

Deploying on EKS on AWS.

P.S. : All DB are created by helm chart of skywalking.

[wu-sheng]

We used those every day. I am not sure. Really, you need to look deeply about the logs.

Even today skywalking.apache.org#demo is deployed by that.

[wu-sheng]

The simplest quick start is using showcase. ref to https://skywalking.apache.org/docs/skywalking-showcase/next/readme/#features.

That deploys the backend, database and demo application with Java agents.

TBO, there are no difference between that.

We also run that in e2e as well, https://github.com/apache/skywalking/blob/master/test/e2e-v2/cases/istio/als/e2e.yaml#L50-L66

[kezhenxu94]

We use the helm hook to create the init job, from you description there seems no init job at all. Please check your cluster and Helm whether hook is supported. https://github.com/apache/skywalking-helm/blob/6fe5e6f0d3b7686c6be0457733e825ee68cb9b35/chart/skywalking/templates/oap-init.job.yaml#L29

[kezhenxu94]

Also I assume you are not using our official terraform https://github.com/apache/skywalking-terraform obviously as it doesn't support AWS EKS if I remember correctly. If you can push the manifests and open a PR I may help to debug and hopefully merge that

[Jatin3708]

Hi @kezhenxu94 :

Terraform module provided is like install of whole infra along with installation of skywalking, while as per the requirement, helm chart needs to be install on EKS in AWS Cloud with help of terraform resource helm_release.

I have used repo : oci://registry-1.docker.io/apache/skywaling-helm with version 4.7.0

My EKS Version is 1.29, while terraform helm provider version is 2.8.0 but not able to link where i can have information that this terraform provider supports which helm version.

If you have any insight here, please let me know.

[Jatin3708]

@kezhenxu94 :

For your reference below is the terraform code :

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "2.8.0"
    }
}

provider "helm" {
  kubernetes {
    host                   = local.cluster_endpoint
    cluster_ca_certificate = local.ca_certificate
    token                  = data.aws_eks_cluster_auth.this.token
  }
  experiments {
    manifest = false
  }
}

resource "helm_release" "skywalking" {
  name             = "skywalking"
  repository       = "oci://registry-1.docker.io/apache/"
  chart            = "skywalking-helm"
  namespace        = "skywalking"
  version          = "4.7.0"
  create_namespace = true
  timeout          = 300
  force_update     = true
  recreate_pods    = true
  max_history      = 10
  wait             = true
  wait_for_jobs    = true
  lint             = true
  cleanup_on_fail  = true

  set {
    name  = "oap.image.tag"
    value = "10.0.0 "
  }
  set {
    name  = "oap.storageType"
    value = "postgresql"
  }
  set {
    name  = "ui.image.tag"
    value = "10.0.0"
  }
  set {
    name  = "elasticsearch.enabled"
    value = "false"
  }
  set {
    name  = "postgresql.enabled"
    value = "true"
  }
}

[kezhenxu94]

hashicorp/terraform-provider-helm#683

[Jatin3708]

Hi @kezhenxu94 :

You are really a savior for me.

This solution work like a charm and on normal troubleshooting way, this was almost impossible to think on this side that this configuration value can also create problem.

Really really thank you so much 💯