Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

observability: annotate Session+SessionPool events #1207

Open
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

odeke-em
Copy link
Contributor

This change adds annotations for session and session pool events to aid customers in debugging latency issues with session pool malevolence and also for maintainers to figure out which session pool type is the most appropriate.

Updates #1170

@odeke-em odeke-em requested review from a team as code owners September 19, 2024 03:48
@product-auto-label product-auto-label bot added the size: m Pull request size is medium. label Sep 19, 2024
@product-auto-label product-auto-label bot added the api: spanner Issues related to the googleapis/python-spanner API. label Sep 19, 2024
@odeke-em odeke-em force-pushed the trace-annotation-SessionPools branch 3 times, most recently from 8d6f2b7 to 4bc937e Compare September 19, 2024 07:37
@odeke-em
Copy link
Contributor Author

Exhibit

On running examples/grpc_instrumentation_enabled.py we get back these traces
Screenshot 2024-09-18 at 9 39 29 PM
Screenshot 2024-09-18 at 9 39 47 PM
Screenshot 2024-09-18 at 9 40 01 PM
Screenshot 2024-09-18 at 9 40 25 PM

@odeke-em
Copy link
Contributor Author

cc @harshachinta

@harshachinta harshachinta added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 19, 2024
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 19, 2024
@harshachinta harshachinta added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 19, 2024
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 19, 2024
@odeke-em odeke-em force-pushed the trace-annotation-SessionPools branch 2 times, most recently from f71d05c to eb9dd5a Compare September 20, 2024 01:27
@harshachinta harshachinta added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 20, 2024
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 20, 2024
@odeke-em
Copy link
Contributor Author

Kindly cc-ing you @harshachinta.

@harshachinta harshachinta added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 23, 2024
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 23, 2024
google/cloud/spanner_v1/pool.py Outdated Show resolved Hide resolved
google/cloud/spanner_v1/_opentelemetry_tracing.py Outdated Show resolved Hide resolved
google/cloud/spanner_v1/pool.py Show resolved Hide resolved
google/cloud/spanner_v1/pool.py Show resolved Hide resolved
google/cloud/spanner_v1/pool.py Outdated Show resolved Hide resolved
google/cloud/spanner_v1/pool.py Outdated Show resolved Hide resolved
start_time = time.time()
current_span = get_current_span()
current_span.add_event(
"Waiting for a session to become available", {"kind": "pinging_pool"}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this not "Acquiring a session"? Lets maintain span consistency across all the pools.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because really it is waiting for a session to become available per

def get(self, timeout=None):
"""Check a session out from the pool.
:type timeout: int
:param timeout: seconds to block waiting for an available session
:rtype: :class:`~google.cloud.spanner_v1.session.Session`
:returns: an existing session from the pool, or a newly-created
session.
:raises: :exc:`queue.Empty` if the queue is empty.
"""
if timeout is None:
timeout = self.default_timeout
session = self._sessions.get(block=True, timeout=timeout)
if not session.exists():
session = self._database.session()
session.create()
return session

and in our document requirements there is a clear need to add that case
Screenshot 2024-10-05 at 10 43 59 PM

google/cloud/spanner_v1/session.py Outdated Show resolved Hide resolved
google/cloud/spanner_v1/session.py Outdated Show resolved Hide resolved
google/cloud/spanner_v1/session.py Outdated Show resolved Hide resolved
@odeke-em
Copy link
Contributor Author

odeke-em commented Oct 7, 2024

Kindly please take another look @harshachinta, feedback addressed, thank you for the code review!

)

if requested_session_count > 0:
current_span.add_event(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

current_span is not initialized.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sheesh, sorry I had a bunch of benchmarks running locally and they interrupted my nox -s unit-3.8 tests so didn't see those but all fixed now.

@harshachinta harshachinta added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 14, 2024
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 14, 2024
@odeke-em odeke-em force-pushed the trace-annotation-SessionPools branch from 73250ca to 0379012 Compare November 14, 2024 18:27
@odeke-em
Copy link
Contributor Author

@harshachinta @surbhigarg92 @sakthivelmanii kindly help me trigger the bots for this PR too. Thank you.

@sakthivelmanii sakthivelmanii added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 14, 2024
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 14, 2024
@odeke-em odeke-em force-pushed the trace-annotation-SessionPools branch from ef7052e to 2d4cbca Compare November 15, 2024 04:58
@odeke-em odeke-em force-pushed the trace-annotation-SessionPools branch from 2d4cbca to 47d6e17 Compare November 15, 2024 05:13
@odeke-em
Copy link
Contributor Author

This PR could use some bots runs. Thank you!

1 similar comment
@odeke-em
Copy link
Contributor Author

This PR could use some bots runs. Thank you!

@harshachinta harshachinta added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 15, 2024
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner Issues related to the googleapis/python-spanner API. size: l Pull request size is large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants