Unauthorised-telemetry and custodianOrgId session expired

[daveanvit]

We ran post-install-script, all steps got executed, user, org, forms etc. got created, but we are not able to login as telemetry is showing unauthorised and forms are also not loading, attaching screenshots, do let me know if any logs required for debugging .

custodianOrg auth error

Any help will be appreciated

[beepdot]

It looks like an API token issue. Can you check in the configmap named player-config has the bearer tokens defined? These are the variables you should check

sunbird_loggedin_register_token
sunbird_logged_default_token
sunbird_anonymous_default_token
sunbird_anonymous_register_token
sunbird_api_auth_token

Please check if all these variables have a value defined?

[daveanvit]

@amit-tarento @keshavprasadms in below block you can see there mismatch of version json file for user and org is this creating an issue?
ESmapping job release tag:- release-4.6.0_RC4

root@sunbird-jenkins:/var/lib/jenkins/workspace/OpsAdministration/dev/Core/PostInstallScript/ansible/roles/es-mapping/files/indices# ls
announcement.json       cbatch.json          compositesearch.json  orgv2.json         usernotes.json
announcementtype.json   cbatchstats.json     course-batch.json     user-courses.json  userprofilevisibility.json
badgeassociations.json  cert-templates.json  location.json         usercourses.json   userv3.json
cbatch-assessment.json  certreg.json         metrics.json          userfeed.json

for user as well it is creating index as user_alias

[amitec006]

@daveanvit thats ok. user and org can have different version

[daveanvit]

@amit-tarento @keshavprasadms we deleted org_alias and created orgv2 index,mapping and added org records as well, but we are still facing same issue, on deleting org_alias we are getting index not found.
can you please help us with all the steps need to follow to create index and mapping from sunbird, and also if there are any jobs that needs to be rebuilt.
Learner pod logs:-

{"timestamp":"2022-09-22T10:25:06.913Z","msg":"{\"eid\":\"LOG\",\"ets\":1663842306912,\"ver\":\"3.0\",\"mid\":\"5f8ff3e2bc76f34d71fc87aeff5af0a3\",\"actor\":{\"id\":\"internal\",\"type\":\"Consumer\"},\"context\":{\"channel\":\"0136282644834385920\",\"pdata\":{\"id\":\"dev.sunbird.learning.service\",\"pid\":\"learner-service\",\"ver\":\"4.6.0\"},\"env\":\"Miscellaneous\",\"cdata\":[{\"id\":\"5f8ff3e2bc76f34d71fc87aeff5af0a3\",\"type\":\"Request\"}],\"rollup\":{}},\"edata\":{\"level\":\"info\",\"type\":\"Api_access\",\"message\":\"{eid='LOG', edata={level=trace, requestid=5f8ff3e2bc76f34d71fc87aeff5af0a3, type=system, message=ENTRY LOG: method : POST, url: /v2/org/search , For Operation : orgSearchV2, params=[{requestedBy=null, filters={isTenant=true, slug=ntp}, id=null, userId=null}]}}\",\"params\":[]}}","lname":"TelemetryEventLogger","tname":"application-akka.actor.default-dispatcher-3","level":"INFO","HOSTNAME":"learner-5597b68c96-m466c","application.home":"/home/sunbird/learner/learning-service-1.0-SNAPSHOT"}
{"timestamp":"2022-09-22T10:25:06.916Z","msg":"{\"eid\":\"LOG\",\"ets\":1663842306915,\"ver\":\"3.0\",\"mid\":\"5f8ff3e2bc76f34d71fc87aeff5af0a3\",\"actor\":{\"id\":\"internal\",\"type\":\"Consumer\"},\"context\":{\"channel\":\"0136282644834385920\",\"pdata\":{\"id\":\"dev.sunbird.learning.service\",\"pid\":\"learner-service\",\"ver\":\"4.6.0\"},\"env\":\"Miscellaneous\",\"cdata\":[{\"id\":\"5f8ff3e2bc76f34d71fc87aeff5af0a3\",\"type\":\"Request\"}],\"rollup\":{}},\"edata\":{\"level\":\"info\",\"type\":\"Api_access\",\"message\":\"ElasticSearchRestHighImpl:search: calling search for index org_alias, with query = {\\\"from\\\":0,\\\"size\\\":250,\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"isTenant.raw\\\":{\\\"value\\\":true,\\\"boost\\\":1.0}}},{\\\"term\\\":{\\\"slug.raw\\\":{\\\"value\\\":\\\"ntp\\\",\\\"boost\\\":1.0}}}],\\\"adjust_pure_negative\\\":true,\\\"boost\\\":1.0}},\\\"_source\\\":{\\\"includes\\\":[],\\\"excludes\\\":[]}}\",\"params\":[]}}","lname":"TelemetryEventLogger","tname":"application-akka.actor.most-used-one-dispatcher-85","level":"INFO","HOSTNAME":"learner-5597b68c96-m466c","application.home":"/home/sunbird/learner/learning-service-1.0-SNAPSHOT"}
{"timestamp":"2022-09-22T10:25:06.923Z","msg":"SearchTelemetryGenerator:generateTelemetry: Error occured in generating Telemetry for orgSearch  ","lname":"org.sunbird.util.search.SearchTelemetryGenerator","tname":"application-akka.actor.brr-usr-dispatcher-126","level":"ERROR","stack_trace":"<#2489e0a6> org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index]\n\tat org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)\n\tat org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2061)\n\tat org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2031)\n\tat org.elasticsearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:1948)\n\tat org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:857)\n\tat org.elasticsearch.client.RestClient$1.completed(RestClient.java:560)\n\tat org.elasticsearch.client.RestClient$1.completed(RestClient.java:537)\n\tat org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)\n\tat org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)\n\tat org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)\n\tat org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)\n\tat org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)\n\tat org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)\n\tat org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)\n\tat org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)\n\tat org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)\n\tat org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)\n\tat org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)\n\tat org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)\n\tat org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)\n\t...\n","HOSTNAME":"learner-5597b68c96-m466c","application.home":"/home/sunbird/learner/learning-service-1.0-SNAPSHOT"}
{"timestamp":"2022-09-22T10:25:06.924Z","msg":"{\"eid\":\"ERROR\",\"ets\":1663842306922,\"ver\":\"3.0\",\"mid\":\"5f8ff3e2bc76f34d71fc87aeff5af0a3\",\"actor\":{\"id\":\"internal\",\"type\":\"Consumer\"},\"context\":{\"channel\":\"0136282644834385920\",\"pdata\":{\"id\":\"dev.sunbird.learning.service\",\"pid\":\"learner-service\",\"ver\":\"4.6.0\"},\"env\":\"Miscellaneous\",\"cdata\":[{\"id\":\"5f8ff3e2bc76f34d71fc87aeff5af0a3\",\"type\":\"Request\"}],\"rollup\":{}},\"edata\":{\"err\":\"INTERNAL_ERROR\",\"stacktrace\":\"ElasticSearchRestHighImpl:search: method Failed with error : org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2061)org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2031)org.elasticsearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:1948)org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:857)org.elasticsearch.client.RestClient$1.completed(RestClient.java:560)org.elasticsearch.client.RestClient$1.completed(RestClient.java:537)org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)java.base/java.lang.Thread.run(Thread.java:829)\",\"errtype\":\"api_access\",\"requestid\":\"5f8ff3e2bc76f34d71fc87aeff5af0a3\"}}","lname":"TelemetryEventLogger","tname":"I/O dispatcher 1","level":"INFO","HOSTNAME":"learner-5597b68c96-m466c","application.home":"/home/sunbird/learner/learning-service-1.0-SNAPSHOT"}
{"timestamp":"2022-09-22T10:25:06.930Z","msg":"Cannot invoke the action","lname":"play.core.server.netty.PlayRequestHandler","tname":"application-akka.actor.default-dispatcher-3","level":"ERROR","stack_trace":"<#2489e0a6> org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index]\n\tat org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)\n\tat org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2061)\n\tat org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2031)\n\tat org.elasticsearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:1948)\n\tat org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:857)\n\tat org.elasticsearch.client.RestClient$1.completed(RestClient.java:560)\n\tat org.elasticsearch.client.RestClient$1.completed(RestClient.java:537)\n\tat org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)\n\tat org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)\n\tat org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)\n\tat org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)\n\tat org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)\n\tat org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)\n\tat org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)\n\tat org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)\n\tat org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)\n\tat org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)\n\tat org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)\n\tat org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)\n\tat org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)\n\t...\n","HOSTNAME":"learner-5597b68c96-m466c","application.home":"/home/sunbird/learner/learning-service-1.0-SNAPSHOT"}
{"timestamp":"2022-09-22T10:25:06.932Z","msg":"Global: onError called for path = /v2/org/search, headers = {Accept=[*/*], Accept-Encoding=[gzip, deflate], Accept-Language=[en-GB,en;q=0.9], Connection=[keep-alive], Content-Length=[54], Content-Type=[application/json], Cookie=[connect.sid=s%3AvpJvqUbcDvwD6j61KR6v4wAeQXNlDOmN.bXLLhhEH8tl490ylcbG95BoZwUdAaui4xOJKfuUfdh4], Host=[learner-service:9000], Origin=[http://sunbird.searceinc.net], Referer=[http://sunbird.searceinc.net/auth/realms/sunbird/protocol/openid-connect/auth?client_id=portal&state=acbdcb3d-d50c-4ed8-81ce-e8ac96c7434e&redirect_uri=http%3A%2F%2Fsunbird.searceinc.net%2Fresources%3Fauth_callback%3D1&scope=openid&response_type=code&version=4], User-Agent=[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36], X-Forwarded-For=[103.238.106.48, 172.26.3.45], X-Forwarded-Host=[sunbird.searceinc.net], X-Forwarded-Port=[8000], X-Forwarded-Proto=[http], X-Real-IP=[103.238.106.48], X-Request-ID=[5f8ff3e2bc76f34d71fc87aeff5af0a3], X-Requested-With=[XMLHttpRequest], X-Scheme=[http]}","lname":"modules.ErrorHandler","tname":"application-akka.actor.default-dispatcher-3","level":"ERROR","stack_trace":"<#2489e0a6> org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=index_not_found_exception, reason=no such index]\n\tat org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)\n\tat org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2061)\n\tat org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2031)\n\tat org.elasticsearch.client.RestHighLevelClient$1.onFailure(RestHighLevelClient.java:1948)\n\tat org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onDefinitiveFailure(RestClient.java:857)\n\tat org.elasticsearch.client.RestClient$1.completed(RestClient.java:560)\n\tat org.elasticsearch.client.RestClient$1.completed(RestClient.java:537)\n\tat org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)\n\tat org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)\n\tat org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)\n\tat org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)\n\tat org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)\n\tat org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)\n\tat org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)\n\tat org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)\n\tat org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)\n\tat org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)\n\tat org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)\n\tat org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)\n\tat org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)\n\t...\n","HOSTNAME":"learner-5597b68c96-m466c","application.home":"/home/sunbird/learner/learning-service-1.0-SNAPSHOT"}

[beepdot]

This is how the index should look like. Ignore the multiple versions in below command. You will have only one version

curl -sS localhost:9200/_cat/indices | egrep "userv|org"
yellow open userv3                             fFHTK-NST-WJnZCqLxBooA 5 1  430349  95743 649.9mb 649.9mb
yellow open userv1                             yx1Ljl03QL2wTXneD9rOYQ 5 1  237733  54497 655.5mb 655.5mb
yellow open org                                REn3D13hTQugO0qlOCY6Ng 5 1  327454      8   4.7gb   4.7gb
yellow open userv2                             WJqHImjAQIeilbXjlUqONg 5 1  430187  82205 696.4mb 696.4mb
yellow open orgv2                              yYwAyqcWSvawoOQBG2adsA 5 1  327595      4 248.8mb 248.8mb

And the alias I was referring should be present in the index, not ansible folder or files in deployment.

curl -sS localhost:9200/orgv2 | jq .orgv2.aliases
{
  "org_alias": {}
}

[beepdot]

Why are you creating them manually? Doesn't the jenkins job create it for you? What are the options you have selected in jenkins jobs?

[sharathkashyap]

@daveanvit Will be locking the thread due to inactivity.Kindly open a seoarate thread if the issue is not resolved