Skip to content

Persistent cache doesn't work for GPU/TPU #8930

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
lsy323 opened this issue Apr 3, 2025 · 1 comment
Open

Persistent cache doesn't work for GPU/TPU #8930

lsy323 opened this issue Apr 3, 2025 · 1 comment
Labels
bug Something isn't working build Build process related matters (e.g. build system). CI CI related change xla:gpu xla:tpu TPU specific issues and PRs

Comments

@lsy323
Copy link
Collaborator

lsy323 commented Apr 3, 2025

Persistent cache test is failing on GPU, with XLA pin update in #8908. The cached computation cannot be deserialized.

[ RUN      ] PersistentCacheTest.test_persistent_cache_mp
E0402 19:38:44.492582203   21190 server_chttp2.cc:40]        ***"created":"@1743622724.492561846","description":"Only 1 addresses added out of total 2 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":404,"referenced_errors":[***"created":"@1743622724.492558996","description":"Address family not supported by protocol","errno":97,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":420,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::1]:8547"***]***
E0402 19:38:58.918947906   22327 server_chttp2.cc:40]        ***"created":"@1743622738.918928838","description":"Only 1 addresses added out of total 2 resolved","file":"external/com_github_grpc_grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":404,"referenced_errors":[***"created":"@1743622738.918926006","description":"Address family not supported by protocol","errno":97,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":420,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::1]:8547"***]***
2025-04-02 19:39:05.879197: W torch_xla/csrc/runtime/pjrt_computation_client.cc:680] Failed to deserialize executable: UNIMPLEMENTED: Deserializing serialized executable not supported.
2025-04-02 19:39:05.880553: W torch_xla/csrc/runtime/pjrt_computation_client.cc:680] Failed to deserialize executable: UNIMPLEMENTED: Deserializing serialized executable not supported.
2025-04-02 19:39:05.887184: W torch_xla/csrc/runtime/pjrt_computation_client.cc:680] Failed to deserialize executable: UNIMPLEMENTED: Deserializing serialized executable not supported.
2025-04-02 19:39:05.892825: W torch_xla/csrc/runtime/pjrt_computation_client.cc:680] Failed to deserialize executable: UNIMPLEMENTED: Deserializing serialized executable not supported.
@lsy323 lsy323 added the xla:gpu label Apr 3, 2025
@ysiraichi ysiraichi added CI CI related change build Build process related matters (e.g. build system). bug Something isn't working labels Apr 7, 2025
@lsy323 lsy323 changed the title [XLA:GPU] Persistent cache doesn't work for GPU Persistent cache doesn't work for GPU/TPU Apr 8, 2025
@lsy323 lsy323 added the xla:tpu TPU specific issues and PRs label Apr 8, 2025
@lsy323
Copy link
Collaborator Author

lsy323 commented Apr 8, 2025

Persistent cache doesn't work for TPU either. It wasn't captured earlier since the test is not run in TPU CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working build Build process related matters (e.g. build system). CI CI related change xla:gpu xla:tpu TPU specific issues and PRs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ysiraichi @lsy323