Skip to content

Conversation

jeremyyeo
Copy link
Contributor

resolves #1280

Problem

See #1280 but GCP Log Explorer doesn't appear to record the invocation id for each log entry - so user cannot be sure of which exact dbt invocation caused the log entry.

Solution

Inject invocation_id into user agent string as the user agent string appears in each and every log entry. Not too sure of the format of the string but this does work currently:

$ dbt run
03:42:27  Running with dbt=1.11.0-a1
03:42:29  Registered adapter: bigquery=1.11.0-a1
03:42:29  Found 1 model, 497 macros
03:42:29  
03:42:29  Concurrency: 4 threads (target='bq')
03:42:29  
03:42:32  1 of 1 START sql table model dbt_jyeo.foo ...................................... [RUN]
03:42:37  1 of 1 OK created sql table model dbt_jyeo.foo ................................. [CREATE TABLE (1.0 rows, 0 processed) in 4.36s]
03:42:37  
03:42:37  Finished running 1 table model in 0 hours 0 minutes and 7.50 seconds (7.50s).
03:42:37  
03:42:37  Completed successfully
03:42:37  
03:42:37  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=1

Then double check the GCP UI to make sure the invocation id the same:

image image

^ Each log entry has

dbt-bigquery-1.11.0a1 dbt-invocation-id/0524e3ce-e7dd-49b9-a5d4-3957f926c043 gl-python/3.11.9 grpc/1.74.0 gax/2.25.1 gapic/3.36.0 gccl/3.36.0,gzip(gfe)

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

Not 100% sure about the test - just tweaked the existing regex.

@cla-bot cla-bot bot added the cla:yes The PR author has signed the CLA label Sep 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla:yes The PR author has signed the CLA
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Inject invocation_id into user_agent for BigQuery for better tracking in Logs Explorer
1 participant