feat: add logfire.url_from_eval(report) method

[Kludex]

Summary

• Adds logfire.url_from_eval(report) method that generates a Logfire dashboard link to view a pydantic_evals.EvaluationReport
• Stores project_url on LogfireConfig from credentials (file or token validation)
• Constructs URL as {project_url}/evals/compare?experiment={trace_id}-{span_id}, returning None if any piece is missing
• Adds pydantic-evals as a dev dependency for type checking

Test plan

• make typecheck — 0 errors
• make lint — all checks passed
• uv run pytest tests/test_url_from_eval.py — 5 tests covering happy path and all None cases
• uv run pytest tests/test_logfire_api.py — logfire-api shim tests pass
• Pre-commit hooks pass

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying logfire-docs with    Cloudflare Pages

Latest commit:	0b2ad3e
Status:	✅  Deploy successful!
Preview URL:	https://ddd3cf9b.logfire-docs.pages.dev
Branch Preview URL:	https://feat-url-from-eval.logfire-docs.pages.dev

View logs

[Copilot]

Pull request overview

This PR introduces a new logfire.url_from_eval(report) helper to generate a Logfire dashboard link for a pydantic_evals.EvaluationReport, and wires config support to retain the project’s dashboard URL derived from credentials/token validation.

Changes:

• Added Logfire.url_from_eval() and exported it at the top-level logfire.url_from_eval.
• Persisted project_url on LogfireConfig when loading credentials / validating tokens.
• Added tests for URL generation and updated dev dependencies/lockfile to include pydantic-evals.

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
logfire/_internal/main.py	Adds Logfire.url_from_eval() URL construction logic.
logfire/_internal/config.py	Stores project_url on config from creds/token validation.
logfire/__init__.py	Re-exports url_from_eval and includes it in __all__.
logfire-api/logfire_api/__init__.py	Exposes url_from_eval via the logfire-api runtime shim.
tests/test_url_from_eval.py	Adds unit tests for URL generation and None cases.
tests/test_logfire_api.py	Verifies url_from_eval is present in the logfire-api shim surface.
pyproject.toml	Adds pydantic-evals to dev dependencies.
uv.lock	Locks pydantic-evals and updates related dependency versions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

project_url may come from credentials and could plausibly include a trailing / (e.g. if provided by a user or backend). Appending /{...} unconditionally can yield a double slash in the final URL. Consider normalizing with project_url = project_url.rstrip('/') before building the path (and ideally use urllib.parse.urlencode for the query string).

[devin-ai-integration]

Devin Review found 2 potential issues.

View 2 additional findings in Devin Review.

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 4 additional findings in Devin Review.

[devin-ai-integration]

Devin Review found 2 new potential issues.

View 8 additional findings in Devin Review.

[devin-ai-integration]

🟡 Stale project_url race when configure() is called multiple times due to orphaned background thread

When configure() is called a second time, _load_configuration() resets self._check_tokens_thread = None and self.project_url = None without joining the previous background thread. The orphaned thread from the first configure() can still write a stale project_url after the reset, and because self.project_url or ... is used, the new thread will then see it as already set and won't overwrite it.

Root cause and scenario

1. logfire.configure(token='token-A') → _load_configuration resets state, initialize() starts thread T1 which calls self._initialize_credentials_from_token('token-A') (an HTTP request)
2. Before T1 finishes, logfire.configure(token='token-B') is called → _load_configuration() at logfire/_internal/config.py:699-700 resets self.project_url = None and self._check_tokens_thread = None (losing the T1 reference), then initialize() starts thread T2 for token-B
3. T1 finishes its HTTP request and executes self.project_url = self.project_url or validated_credentials.project_url at logfire/_internal/config.py:1004, setting project_url to project-A's URL
4. T2 finishes and runs the same line, but since self.project_url is already set (by T1), the or short-circuits and project-B's URL is never stored
5. url_from_eval() at logfire/_internal/main.py:891 calls wait_for_token_validation() which only joins T2 (T1's reference was lost), reads self.project_url, and returns a URL pointing to the wrong project

Impact: url_from_eval could return a URL for a previous project configuration's dashboard instead of the current one. This is limited to the uncommon case of calling configure() multiple times with different tokens where the first token validation thread hasn't finished.

Prompt for agents

In _load_configuration (logfire/_internal/config.py around line 699), before resetting self._check_tokens_thread = None, join the old thread (if it exists) to ensure it has finished writing. This prevents the orphaned thread from writing a stale project_url after the reset. For example:

    self.additional_span_processors = additional_span_processors
    self.project_url: str | None = None
    if hasattr(self, '_check_tokens_thread') and self._check_tokens_thread is not None:
        self._check_tokens_thread.join(timeout=0)  # non-blocking, just try
    self._check_tokens_thread: Thread | None = None

Alternatively, a more robust approach is to use a generation counter: increment a counter in _load_configuration, capture it in the check_tokens closure, and only write self.project_url if the generation still matches. This avoids blocking configure() on an HTTP request.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[dmontagu]

I feel like we need a place to put functions that are specific to our web platform, ideally grouped a bit more by function. There are going to be variables soon, then datasets, and more to come. I feel like we should be a bit thoughtful about where the APIs go rather than just haphazardly adding methods to the logfire instance. But 🤷 if it unblocks stuff it's probably worth biasing toward action