Could not find variable in saved models

[yashprakashsharma]

Here are references for the issue tensorflow/tensorflow#73158 and tensorflow/tensorflow#72963

[yashprakashsharma]

also getting issue from cifar10_model and iris_model:

logs: 
2024-08-08 08:16:11.407139: I external/org_tensorflow/tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-08-08 08:16:11.500499: I tensorflow_serving/model_servers/server.cc:77] Building single TensorFlow model file config:  model_name: cifar10_model model_base_path: /models/cifar10_model
2024-08-08 08:16:11.503247: I tensorflow_serving/model_servers/server_core.cc:474] Adding/updating models.
2024-08-08 08:16:11.503278: I tensorflow_serving/model_servers/server_core.cc:603]  (Re-)adding model: cifar10_model
2024-08-08 08:16:11.709794: I tensorflow_serving/core/basic_manager.cc:740] Successfully reserved resources to load servable {name: cifar10_model version: 1}
2024-08-08 08:16:11.709855: I tensorflow_serving/core/loader_harness.cc:68] Approving load for servable version {name: cifar10_model version: 1}
2024-08-08 08:16:11.709884: I tensorflow_serving/core/loader_harness.cc:76] Loading servable version {name: cifar10_model version: 1}
2024-08-08 08:16:11.716917: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: /models/cifar10_model/1
2024-08-08 08:16:11.729105: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:52] Reading meta graph with tags { serve }
2024-08-08 08:16:11.729159: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:147] Reading SavedModel debug info (if present) from: /models/cifar10_model/1
2024-08-08 08:16:11.730837: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-08-08 08:16:11.775236: I external/org_tensorflow/tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:388] MLIR V1 optimization pass is not enabled
2024-08-08 08:16:11.782870: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:236] Restoring SavedModel bundle.
2024-08-08 08:16:11.948435: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:220] Running initialization op on SavedModel bundle at path: /models/cifar10_model/1
2024-08-08 08:16:11.961415: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:462] SavedModel load for tags { serve }; Status: success: OK. Took 244832 microseconds.
2024-08-08 08:16:11.963256: I tensorflow_serving/servables/tensorflow/saved_model_warmup_util.cc:82] No warmup data file found at /models/cifar10_model/1/assets.extra/tf_serving_warmup_requests
2024-08-08 08:16:12.070870: I tensorflow_serving/core/loader_harness.cc:97] Successfully loaded servable version {name: cifar10_model version: 1}
2024-08-08 08:16:12.073985: I tensorflow_serving/model_servers/server_core.cc:495] Finished adding/updating models
2024-08-08 08:16:12.074074: I tensorflow_serving/model_servers/server.cc:121] Using InsecureServerCredentials
2024-08-08 08:16:12.074608: I tensorflow_serving/model_servers/server.cc:388] Profiler service is enabled
2024-08-08 08:16:12.079460: I tensorflow_serving/model_servers/server.cc:423] Running gRPC ModelServer at 0.0.0.0:8500 ...
[warn] getaddrinfo: address family for nodename not supported
2024-08-08 08:16:12.083437: I tensorflow_serving/model_servers/server.cc:444] Exporting HTTP/REST API at:localhost:8501 ...
[evhttp_server.cc : 250] NET_LOG: Entering the event loop ...
2024-08-08 08:17:38.815509: I external/org_tensorflow/tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: FAILED_PRECONDITION: Could not find variable sequential/conv2d/kernel. This could mean that the variable has been deleted. In TF1, it can also mean the variable is uninitialized. Debug info: container=localhost, status error message=Resource localhost/sequential/conv2d/kernel/N10tensorflow3VarE does not exist.
        [[{{function_node __inference_serving_default_45404}}{{node sequential_1/conv2d_1/convolution/ReadVariableOp}}]]
2024-08-08 08:17:38.815626: I external/org_tensorflow/tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: ABORTED: Stopping remaining executors.
2024-08-08 08:17:45.111537: I external/org_tensorflow/tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: ABORTED: Stopping remaining executors.

[nkovela1]

HI @yashprakashsharma, this appears to be a duplicate of #20095 given the issue you referenced above. Closing the issue for now to avoid duplication, thanks!

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No