Deploy -> Kubernetes -> Logging (oauth2-proxy)

[monojit18]

The Deploy is failing
Branch: release-4.10.1_RC1
Note: Only failing with OAuth2-Proxy option; flunetd, Kkibana Passed thru well

============================================================================

TASK [logging : helm install] **************************************************
fatal: [localhost]: FAILED! => {
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m --force oauth2-proxy /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/kubernetes/helm_charts/logging/oauth2-proxy -n logging -f /tmp/oauth2-proxy.yaml",
"delta": "0:10:01.239261",
"end": "2022-09-13 20:01:17.177161",
"rc": 1,
"start": "2022-09-13 19:51:15.937900"
}

STDOUT:

Release "oauth2-proxy" does not exist. Installing it now.

STDERR:

coalesce.go:196: warning: cannot overwrite table with non table for alphaConfigFile (map[])
Error: release oauth2-proxy failed, and has been uninstalled due to atomic being set: timed out waiting for the condition

============================================================================================

[beepdot]

When the deployment job is running, are you able to see the pods being created in logging namespace? Any errors in the pod logs of oauth proxy?

[daveanvit]

@keshavprasadms I am also getting same issue for kibna, pod is not healthy for me it was healthy earlier
pod logs:

{"type":"log","@timestamp":"2022-09-15T09:44:36Z","tags":["license","warning","xpack"],"pid":1,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [invalid_index_name_exception] Invalid index name [_xpack], must not start with '_'., with { index_uuid=\"_na_\" & index=\"_xpack\" } :: {\"path\":\"/_xpack\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"invalid_index_name_exception\\\",\\\"reason\\\":\\\"Invalid index name [_xpack], must not start with '_'.\\\",\\\"index_uuid\\\":\\\"_na_\\\",\\\"index\\\":\\\"_xpack\\\"}],\\\"type\\\":\\\"invalid_index_name_exception\\\",\\\"reason\\\":\\\"Invalid index name [_xpack], must not start with '_'.\\\",\\\"index_uuid\\\":\\\"_na_\\\",\\\"index\\\":\\\"_xpack\\\"},\\\"status\\\":400}\"}"}
{"type":"log","@timestamp":"2022-09-15T09:44:39Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [illegal_argument_exception] request [/_template/.kibana_task_manager] contains unrecognized parameter: [include_type_name]"}
{"type":"log","@timestamp":"2022-09-15T09:44:42Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [illegal_argument_exception] request [/_template/.kibana_task_manager] contains unrecognized parameter: [include_type_name]"}
{"type":"log","@timestamp":"2022-09-15T09:44:45Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [illegal_argument_exception] request [/_template/.kibana_task_manager] contains unrecognized parameter: [include_type_name]"}
{"type":"log","@timestamp":"2022-09-15T09:44:48Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [illegal_argument_exception] request [/_template/.kibana_task_manager] contains unrecognized parameter: [include_type_name]"}
{"type":"log","@timestamp":"2022-09-15T09:44:51Z","tags":["warning","task_manager"],"pid":1,"message":"PollError [illegal_argument_exception] request [/_template/.kibana_task_manager] contains unrecognized parameter: [include_type_name]"}

[beepdot]

Is the log elasticsearch cluster up and running? Can you post the elasticsearch version and also the kibana version?

1. Log ES IP - https://github.com/project-sunbird/sunbird-devops/blob/master/private_repo/ansible/inventory/dev/Core/hosts#L12-L13
2. Please post output of below commands
   curl localhost:9200
curl localhost:9200/_cat/indices
3. Please share the image name of the kibana pod / deployment. If the pods are not running now, please deploy it from jenkins and share the image name.

[beepdot]

Please note this thread is discussing about oauth proxy pods. @daveanvit I would like you to create a separate thread for kibana so that other community members can also benefit from the solution.

[sharathkashyap]

@keshavprasadms are we done with this thread

[beepdot]

Yes please close the thread